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Executive Summary 



The collection, analysis and use of educational data are central to the improvement of 
student outcomes envisioned by No Child Left Behind (NCLB). The use of data in educational 
decision making is expected to span all layers of the education system — from the federal to the 
state, district, school and classroom levels. The implementation of the NCLB legislation has been 
accompanied by a demand for data systems capable of providing a longitudinal record of each 
student’s educational experiences and performance over time. 

The national Study of Education Data Systems and Decision Making, sponsored by the 
U.S. Department of Education’s Office of Planning, Evaluation and Policy Development, is 
documenting the availability of education data systems, their characteristics, and the prevalence 
and nature of data- informed decision making in districts and schools.^ The study is examining 
both the implementation of student data systems per se and the broader set of practices involving 
the use of data to improve instruction, regardless of whether or not the data are stored in and 
accessed through an electronic system. 

The conceptual framework developed for the study identifies six prerequisites and supports 
for data-informed decision making: (a) state, district and school data systems; (b) leadership for 
educational improvement and the use of data; (c) tools for generating actionable data; (d) social 
structures and time set aside for analyzing and interpreting data; (e) professional development 
and technical support for data interpretation; and (f) tools for acting on data. 

This conceptual framework inspired the study’s research questions: 

• What kinds of systems are available to support district and school data-driven decision 
making? Within these systems, how prevalent are tools for generating and acting on 
data? 

• How prevalent are organizational supports for school use of data systems to inform 
instruction? 

• How are school staff members using data systems? Do they know how to interpret 
student data? How is school staff members’ use of data systems and of data more 
broadly influencing instruction? 



This interim report describes findings from multiple sources. It draws on case study 
findings from the first round of site visits in nine purposively sampled districts, nominated on the 
basis of the strength of their data use activities. Researchers interviewed district staff members as 
well as principals and teachers from three schools within each district. In addition, a set of 
scenarios involving hypothetical student data were presented to teachers at each school to probe 
their understanding of student data. This report draws also on data from secondary sources 



* The term data-informed decision making is used throughout this report in preference to the more common term 
data-driven decision making in recognition of the fact that few decisions are based wholly on quantitative data. 

^ Information on how districts, schools and respondents were selected for participation in the case study sample can 
be found on pages 6-9 of this report; additional information on the National Educational Technology Trends 
Study (NETTS) survey sample can be found on page 6 and in Appendix A. 
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(spring 2007 district and teacher surveys from the U.S. Department of Edueation’s National 
Edueational Teehnology Trends Study). Using these sourees, the report deseribes the student 
data systems available to sehool staff members, how sehool staff members are using the systems 
and other forms of student data, teaehers’ understanding of data displays and data interpretation 
issues, and the supports and ehallenges for sehool-level use of student data in planning and 
implementing instruetion. 

Early Findings 

The burgeoning activity around student data systems over the last five years is being 
felt at the district and school levels. The dramatie inerease in teaeher aeeess to student data 
systems has been doeumented in the two NETTS teaeher surveys. In ease study sehools and 
distriets, teaehers were reeeiving professional development around data use and were using data 
reports from systems for sehool improvement. Sehool teams working on partieular initiatives, 
sueh as Reading Eirst, were using student data to guide their aetions. 

Data from student data systems are being used in school improvement efforts but are 
having little effect on teachers’ daily instructional decisions as evidenced in case study 
districts. Despite progress in giving teaehers aeeess to student data, it is elear that in many 
distriets, the use of loeally generated data to inform instruetion is an aetivity separate from use of 
data systems eontaining student seores on standardized tests. Distriet and sehool uses of data 
systems to store, organize and report standardized test seores typieally foeus on aeeountability 
eoneerns and on efforts to ensure that loeal eurrieulum and instruetion are well aligned with state 
assessments. Neither the type of assessment for whieh data are available nor the time frame of 
assessment aetivities serves the needs of elassroom teaehers making deeisions on a daily basis. 
Case study sehools did offer evidenee that teaehers and teaeher teams were using data to guide 
elassroom instruetion, but these data generally eame from assessments elosely aligned with loeal 
instruetion, and the data were typieally not stored on the student data system eontaining state 
assessment seores. The integration of elassroom and state assessment data in the same eleetronie 
system is not eommon, even in ease study distriets noted for their data systems and data-using 
eulture. 

More speeifie interim findings address the issues of data system availability, supports 
provided for data-informed deeision making, sehools’ use of data and teaehers’ preparation for 
using data to inform instruetion. 

Data Systems Available to Local Educators 

Although teacher-reported access to student data systems is growing rapidly, the data 
systems themselves are currently not supporting instructional decision making at the school 
level. District data systems often cannot share data across systems, are not user friendly, contain 
limited data, and lack instructional tools to help teachers act on the data provided to them. 

Teacher-reported access to student data systems grew significantly, from 48 percent 
in 2005 to 74 percent in 2007. This growth in teacher access was estimated using national 
estimates based on responses of teachers to the National Educational Technology Trends Study 
(NETTS) teacher survey. 
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Even though teacher access to data systems is growing rapidly, systems often lack the 
kinds of data that teachers find most useful for instructional decision making. Among 
teachers who said they had access to a student data system (Exhibit 1), the data most frequently 
available to these teachers were student attendance data (74 percent) and grades (67 percent). 
Only 55 percent of the teachers with access to a student data system (or 41 percent of all 
teachers) had access to their current students’ performance on benchmark or diagnostic tests. 

Wayman (2005) describes four common types of educational data systems: (a) student 
information systems providing real-time accounting for school functions such as attendance, (b) 
assessment systems supporting the rapid organization and analysis of a wide array of assessment 
data, (c) data warehouses that link multiple databases to provide access to historical data of all 
types, and (d) instructional/curriculum management systems. The NETTS teacher survey data 
suggest that teachers are most likely to have access to the first of these system types. Exhibit 1 
summarizes the survey findings. 



Exhibit 1 

Data Available to Teachers 



Types of Data Available 


Percentage of 
Teachers With 
System Access 


Percentage of All 
Teachers 


Attendance data 


74 


55 


Student grade data 


67 


50 


Performance on benchmark or diagnostic tests 
taken by your current students 


55 


41 


Spring 2006 standardized test scores for the 
students you taught last year (2005-06) 


46 


34 


Spring 2006 standardized test scores for your 
current students 


44 


33 


Fall 2006 standardized test scores for your 
current students 


33 


25 


Multiple years of standardized test scores for 
individual students 


29 


11 


Course enrollment histories for students 


22 


16 


Students’ prior school(s) attended 


16 


12 


Students’ participation in supplementary 
education programs (e.g., tutoring) 


8 


6 



Exhibit reads: Among teachers who had access to an electronic student data system in 2006-07, 74 
percent reported that they had access to attendance data through the data system, which represents 55 
percent of all teachers (with or without access to a data system). 

Source: 2006-07 NETTS teacher survey. 



At one point, proponents of data-informed decision making in education offered the vision 
of a single integrated system combining multiple sources of information. As states and districts 
have gained experience with data systems, most experts have come to agree that it is more 
practical to strive for interoperability among different education data systems so users can move 
data between systems or combine data from different systems easily rather than try to build a 





single education data system that would serve all purposes (Datnow, Park and Wohlstetter 2007; 
Wayman 2005). Interoperability among data systems was an issue within the case study 
districts. Staff members at the nine districts in the spring 2007 case study sample were all using 
two or more separate systems with limited interoperability. None had a fully interoperable set of 
data systems. Typically, the information needed for educational decision making was spread 
across multiple systems without mechanisms for regular transport of information from one 
system to another. As a result, neither teachers nor administrators see a comprehensive record of 
students’ educational experiences and performances that is both longitudinal and up to date. 

Barriers to data system use that teachers cited on the NETTS survey suggest that a 
significant proportion of teachers with access to an electronic student data system have difficulty 
getting the kinds of data they want to see out of the data system.^ Teachers reported feeling 
hampered by their inability to use data queries to get pertinent data from their data systems and 
by the limited utility of the kinds of information available in the student data systems for 
deciding what and how to teach (each reported by 29 percent). Smaller but significant 
proportions of teachers reported that they had trouble finding the information they were looking 
for on the system (24 percent) and that the system was hard to use (20 percent). Among school 
staff members interviewed as part of the case study research, the three most commonly cited 
barriers were (a) lack of training in how to use the data system or to derive instructional 
implications from system data, (b) the lack of time to engage in data exploration and reflection, 
and (c) the weakness of the available data. 

Tools in data systems to help teachers improve decisions about instructional practice 
are not the norm, even in case study districts identified as high data users. Data from the 
case studies suggest that only a minority of data systems incorporate resources such as 
instructional materials, model lesson plans, and formative assessment results linked to 
frameworks and curriculum guides. Only three case study districts had data systems that 
provided such resources. One additional district was developing a system with this capacity. 

Supports for Data-Informed Decision Making 

Effective data-informed decision making requires not only access to useful data but also 
well-designed supports such as leadership to model data use and supported time for reflection on 
data. Although districts are beginning to provide supports such as training and support staff for 
data-informed decision making, there is a great deal of variability in the depth and breadth of 
these supports, even among districts identified as model data users. Eederal programs have 
played an important role in helping teachers examine data and apply this information to 
instructional practice. 

Case study districts demonstrated their support and leadership for schools’ use of 
data through purchasing systems, modeling data use, and providing school-based support 
positions. More specifically, district supports for schools’ use of data included (a) making data- 
informed decision making a priority and purchasing or developing a data system; (b) considering 
ability to use data as a criterion for teacher hiring; (c) providing training and support positions 
for system implementation; (d) initiating districtwide, data-informed decision-making activities 



^ All percentages reported for teacher survey respondents in the remainder of the executive summary are for teachers 
who indicated they had access to an electronic student data system in 2006-07 unless otherwise indicated. 
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in addition to professional development; and (e) using the data system for decision making at the 
district level (e.g., evaluating programs, principals, teachers). 

A number of federal programs have supported schools’ use of data to inform 
instruction. Staff members at case study schools described the use of funds from Title I, 
Enhancing Education Through Technology (EETT) and Reading Eirst to support school-based 
staff members who help teachers examine data and draw implications for instruction. Reading 
Eirst appeared to be particularly effective in supporting data use because Reading Eirst coaches 
could provide teachers with different instructional approaches appropriate for students with 
various patterns of assessment results. 

Leadership for data-informed decision making at the school level can extend beyond 
the principal. Previous case studies of data-informed decision making in schools have stressed 
the importance of leadership by the principal (e.g., setting the context for using student data, 
providing the time for reflection on data, and communicating expectations for teachers’ data 
use). Although these functions are important, the case studies suggest that they may be 
performed by individuals in a variety of job roles; instructional coaches, department lead 
teachers, and instruction and assessment coaches were providing leadership for using data in 
many of the case study schools. 

Districts appear to be responding to the need for professional development for 
teachers that focuses on data analysis skills or data-informed decision making more 
generally. On the 2006-07 NETTS teacher survey, 39 percent of teachers reported that the 
professional development they received about data-informed decision making had prepared them 
to use data to improve student achievement. At the same time, most teachers reported that they 
would benefit from additional professional development on data-informed decision making. All 
case study districts provided some form of districtwide professional development, but it varied in 
terms of (a) who received the training and (b) the training’s content, duration and format. 

Organizational structures that support data use at the school level can include 
designated time for teachers to review and discuss data in small groups, assigned support 
staff, and the adoption of procedures for discussing data. On the 2006-07 NETTS teacher 
survey, most teachers reported having positive perceptions of support for using and interpreting 
data that they could access from data systems. Eor example, a majority of teachers with access to 
a data system (7 1 percent) agreed or agreed strongly that when they needed help making sense of 
the data in district systems, they knew someone who could help. Case study districts provided 
even stronger support for data-informed decision making than the typical level indicated in the 
NETTS teacher survey responses. Six out of nine offered some sort of district- funded, school- 
based staff to support teachers’ data use. In some schools and districts, individuals in these 
positions extracted relevant data from the system and guided teachers in drawing inferences for 
action. In other cases, the coaches’ role involved helping teachers learn to analyze data 
themselves, with coaches motivating teachers to examine data and providing just-in-time 
informal professional development for data interpretation and connecting data to instructional 
strategies. 
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How Schools Are Using Data 

A significant barrier to implementing data-informed decision making is a lack of expertise 
among school staff members in the area of data analysis. Districts and schools are addressing this 
challenge by supporting collaborative activities to discuss student data, but more training is 
required through both pre- service and in-service programs to provide teachers with a full range 
of data literacy skills. 

Most teachers who use a student data system do so not only on their own but also in 
collaboration with colleagues. Among NETTS teacher survey respondents, the most commonly 
reported context for using a student data system was on one’s own (78 percent), followed closely 
by data use with colleagues in a department or grade (71 percent). In the case study districts, the 
most frequently cited groupings for data use were the grade-level team, sometimes facilitated by 
a coach, and all-staff faculty meetings. 

Both teachers and district staff members in the case study sample express concerns 
about teachers’ ability to understand data. The majority of NETTS teacher survey 
respondents also reported that they would benefit from additional professional development on 
data-informed decision making. To better assess teachers’ understanding of data, project staff 
incorporated data scenarios into the case studies, which were presented individually to several 
teachers at each school. The scenarios consisted of hypothetical sets of classroom, school and 
district data as well as a set of questions about the data and its implications for practice. These 
data scenarios were administered to a sample of teachers in each case study school to elicit their 
thinking about data and reveal the concepts and skills that teachers can bring to data-informed 
decision making. An expert panel identified the necessary data skills and concepts, which were 
clustered within five aspects of data-informed decision making: Question Posing, Data Eocation, 
Data Comprehension, Data Interpretation, and Data Use (making instructional decisions based 
on data). 

Teachers’ responses to scenarios concerning hypothetical student data suggest that 
teachers can locate the data they want within complex tables or graphs but often lack other 
data literacy skills. Exhibit 2 presents the average proportion of teachers earning full credit for 
their responses to data scenario items of various types. 

Data Eocation items, which required the literal reading of a table or graph, were easy for 
most teachers. Teachers had some difficulty with Data Comprehension items such as those 
involving multiple data points for both school and district performance over time or where 
proportion and absolute value had to be distinguished. Data Interpretation items, which involved 
drawing inferences from data patterns, also proved difficult. These findings have implications for 
in-service and pre-service teacher education programs. 



xii 




Exhibit 2 

Percentage of Correct Responses, by Item Type Across Schools 



Skill 


Percent Correct 


Sample Item 


Data Location 


81 


What was Oak School’s average 
Total Math Score in 2003-04? 


Data 

Comprehension 


64 


Oak School’s progress in 
narrowing the Grade 4 math 
achievement gap with the rest of 
the district has been in problem 
solving rather than computation. 
(Teacher must agree or disagree 
and explain reasoning.) 


Data 

Interpretation 


48 


This year all African American 
students should get more 
intensive instruction in 
mathematics. (Teacher must 
agree or disagree and explain 
reasoning.) 



Exhibit reads: For the Data Location items, the average percentage of teachers in a school earning 
full credit on the item was 81 percent. 

Note: Question Posing and Data Use categories are not included in the table because they were 
represented by only one scored item each (percentage correct of the related items were 56 percent 
and 87 percent, respectively). 

Source: 2006-07 data scenarios. 






1. Introduction and Approach 



Over the past six years, meeting the data requirements of NCLB and adapting or acquiring 
database systems capable of generating the required student data reports have consumed much of 
the attention of district and state assessment and technology offices. This work has been 
necessary but insufficient for data-informed decision making to influence education. Data- 
informed educational decision making consists of much more than just a data system: it includes 
a set of expectations and practices around the ongoing examination of data to ascertain the 
effectiveness of educational activities to improve outcomes for students. The implementation of 
data-informed decision making cannot occur without leadership and supporting conditions such 
as tools for generating actionable data, professional development and technical support for data 
interpretation, and time set aside for analyzing and interpreting data. For definitions of terms 
used in this report, see Exhibit A-1 in Appendix A. 

To understand the role of data systems and the supports necessary for teachers to use data 
from any source (electronic and nonelectronic) to inform educational practice, the U.S. 
Department of Education’s Office of Planning, Evaluation and Policy Development is 
sponsoring a national study, the Study of Education Data Systems and Decision Making. The 
study is addressing a set of basic questions: 

1 . What kinds of systems are available to support district and school data-driven decision 
making? Within these systems, how prevalent are tools for generating and acting on data? 

2. How prevalent are organizational supports for school use of data systems to inform 
instruction? 

3. How are school staff members using data systems? Do they know how to interpret 
student data? How is school staff members’ use of data systems and of data more broadly 
influencing instruction? 

This report describes interim findings from the first round of data collection and analysis for that 
study. 

Conceptual Framework of Data-informed Decision-Making Process and Supports 

The conceptual framework on which the study design and instrumentation were based is 
presented here as a prelude to the description of the study approach and early findings. 
Proponents of data-informed decision making call on educators to adopt a continuous- 
improvement perspective with an emphasis on goal setting, measurement and feedback loops so 
teachers and administrators can reflect on their programs and processes, relate them to student 
outcomes, and make refinements suggested by the outcome data."^ To make this process 
continuous, educators must then implement the refined practices, again measuring outcomes and 
looking for places for further refinement. An important distinction between this practice and 



It should be noted that in business, a continuous improvement process is feasible under conditions where there is 
standardized implementation of a set of processes and standardized measurement of outcomes. Under these 
conditions, monitoring of outcome measures typically results in small, incremental improvements to the standard 
process. Within schools, standardized implementation of teaching practices is not the norm, so implementing 
continuous improvement processes becomes more difficult. 
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most educational programs or reforms is that this improvement cycle does not end with the first 
set of program refinements — it is a way of thinking and a way of life rather than a discrete event 
or process (Schmoker 1996). 

Exhibit 1-1 shows the stages in a data-informed, continuous -improvement process: plan, 
implement, assess, analyze data and reflect (as a precursor to more planning and a refined 
implementation). As the graphic suggests, components of data-informed decision making are 
part of a continuous cycle. The starting point may vary, and there is no fixed end point. In 
addition, key prerequisites and supporting conditions are essential for this process to be 
successful. 



Exhibit 1-1 

Conceptual Framework for Data-informed Decision Making 




Prerequisites and Supporting Conditions 



1 . Data systems 2. Leadership for 
improvement 
and use of 
data 



3: Tools for 4. Social 

generating structures and 

data time set aside 

for reflection on 
data 



5. Professional 6. Tools for 
development acting on data 
and technical 
support for data 
interpretation 



The Process of Using Data for Improvement 

The stages in this cycle, though often closely linked and overlapping in execution, can be 
characterized as follows: 

Reflect. As educational decision makers review their current practices and outcomes, they 
identify areas in which improvement is needed. A formal needs assessment may be conducted. 
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Plan. In response to concerns about existing practices or outcomes, educational decision 
makers may devise a plan. For example, a district concerned about low or uneven reading levels 
among students in the primary grades may decide to adopt a particular reading curriculum for all 
its primary classrooms. A school may design a “safety net” program for students who scored 
below grade level on the prior year’s mathematics test. Often, a group of teachers or 
administrators will jointly design the new process or materials. 

Implement. Once an intended educational change is planned, it must be implemented 
within the school or classroom. Implementation includes (a) disseminating documentation and 
supporting materials concerning the new approach and (b) training the teachers, administrators or 
other personnel responsible for making it happen. It also includes the decisions that these 
individuals make in deciding how to implement the mandated or agreed-on change within their 
particular school or classroom. 

Assess. An organization dedicated to continuous improvement does not just implement a 
new program or process. The organization builds in opportunities to measure the outcomes 
obtained with the educational change. Often, this effort will include some kind of student 
assessment such as district end-of-unit tests or teachers’ quarterly appraisals of students’ writing 
fluency. Depending on the nature of the innovation, the measured outcomes could be of many 
different types, including, for example, student attendance or parent attitudes toward the school 
as expressed on a survey or through participation in parent- teacher conferences. 

Analyze data. Having accumulated assessment data, educators then need to analyze the 
data in ways that relate the outcomes to processes and turn the data into actionable information. 
Often this analysis includes segmenting the findings by student subgroups or by groupings used 
within the school or classroom (because students in different groups received different services). 
It may also include the analysis of data trends over time and the search for patterns in multiple 
measures related to a given change or issue. 

Reflect. Having analyzed the data, educators then must make sense of observed changes or 
the lack of change. This reflection stage is where participants interpret the findings of the data 
analysis and draw implications for action — that is, where they develop ideas for how they can 
refine and improve their program. The inferences drawn in this stage lay the groundwork for 
developing a new plan. 

Supports for the Process of Using Data for Improvement 

Making the continuous-improvement perspective and the processes of data-informed 
decision making part of the way in which educators function requires a major cultural change. 
Such a change will not occur without leadership, effort and well-designed supports. The bottom 
portion of Exhibit 1-1 identifies six major types of prerequisites and supports for data-informed 
decision making: (a) state, district and school data systems; (b) leadership for educational 
improvement and the use of data; (c) tools for generating actionable data; (d) social structures 
and time set aside for analyzing and interpreting data; (e) professional development and technical 
support for data interpretation; and (f) tools for acting on data. The national Study of Education 
Data Systems and Decision Making, part of the National Technology Activities (NTA) task 
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order, is collecting data on the prevalence and quality of student data systems and associated 
supports. The nature of these systems and supports are described below. 

Data systems. NCLB has stimulated an unprecedented level of state activity directed at 
improving education data systems. Federal requirements for reporting schools’ year-to-year 
progress in raising achievement overall and for specific student categories have led to an 
examination of information system adequacy and the adoption or development of new software 
systems in many states. State systems typically include student enrollment information, basic 
demographic data, special program designation (if applicable), and scores on state-mandated 
achievement tests (in most cases, an annual spring testing in language arts and mathematics and 
often a proficiency or “exit” examination required for a high school diploma).^ 

The National Center for Educational Accountability (NCEA) has undertaken a systematic 
review of the characteristics of state-level education data systems. Over the last four years, states 
have made major strides toward putting in place data systems that will support longitudinal 
analysis of student progress. In 2005, for example, 36 states used unique student identification 
numbers statewide so students could be followed if they changed districts. In 2007, 45 states did 
so. As of 2007, a majority of state systems also included student-level enrollment, demographic 
and program participation information; the ability to match individual students’ test scores from 
year to year to measure academic growth; and student-level graduation and dropout data. Eess 
prevalent were the inclusion of scores on college readiness examinations (such as the SAT, ACT 
or Advanced Placement exams); the ability to match students’ Pre-K-12 records with the state’s 
higher education system records; and a data audit system for assessing data quality, validity and 
reliability.^ 

School districts typically maintain their own data systems. In addition to student scores for 
state-mandated tests, which they get from the state or the state-designated vendor, district 
systems often include information about a student’s teachers, grades, attendance, disciplinary 
infractions and scores on district tests. 

In terms of the software applications designed to support access to and analysis of data 
from these systems, most of the commercial products are targeted to districts. If districts have 
developed and implemented districtwide assessments that students take throughout the school 
year, these products typically support school-level access to the assessment results. 

Other software applications are targeted for school use, and these tend to include features 
such as electronic grade books and the capability to incorporate teacher-developed formative 
assessments. Some of these applications have the capability to store examples of student work in 
an “electronic portfolio.” 

In theory, state-, district- and school-level data systems could be interoperable (U.S. 
Department of Education 2004), and there could be “one-stop shopping” for educational data 



^ NCLB requires that states create annual assessments that measure what children know and can do in reading and 
mathematics in grades 3 through 8 and that they test at least once between grades 10 and 12 by 2005-06 for 
reading and mathematics and by 2007-08 for science. 

® See http;//www.dataqualitycampaign.org/survey_results/ for a complete report of NCEA’ s 2007 state survey. 
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coming from sources ranging from the individual school to the U.S. Department of Education 
(what one vendor characterizes as the “secretary to Secretary” solution). Such a set of integrated 
systems does not yet exist, but federal efforts are moving toward uniform and consolidated 
reporting of data to the federal level, and some states are attempting to incorporate features in 
state systems that will make them more useful to districts and schools (Palaich, Good and van 
der Ploeg 2004). 

Leadership for educational improvement and use of data. Pioneering efforts to promote 
data-informed decision making within districts and schools have found that the active promotion 
of the effort on the part of the superintendent or principal is vital (Cromey 2000; Halverson et al. 
2005; Herman and Gribbons 2001; Marsh, Pane and Hamilton 2006). District and school leaders 
need to issue the “call to arms” for improving education and using data as a tool to bring about 
that improvement. Typically, they play a major role in framing targets for educational 
improvement, setting expectations for staff participation in data-informed decision making, and 
making resources such as supported time available to support the enterprise. 

Tools for generating actionable data. Increasingly, student achievement data are 
available at the school level in a form that can be disaggregated by student category (ethnicity, 
free or reduced-price lunch status, special education status and so on). Software systems to 
support data-informed decision making all generate standard student achievement reports, and 
many also produce custom reports for user-designated student groups (an important feature for 
school staff members who want to examine the effects of locally developed services for specific 
student groups). Research indicates, however, that often, school staff members do not find the 
kinds of data these systems provide particularly useful for guiding instruction (Mandinach et al. 
2005; Marsh, Pane and Hamilton 2006). School staff members are frustrated by the fact that the 
data available to them are typically data reflecting performance on a state achievement test taken 
six or more months earlier. Teachers want up-to-date information on their current group of 
students, not the students in the same grade level last year. They want a greater level of detail 
concerning individual students’ strengths and weaknesses than they can get from standardized 
test scores (Thorn 2002). Although far less common than systems that provide data from prior 
testing, there are examples of systems that produce additional information for decision making 
through tools such as formative assessments that students may take online. In addition, some 
system design companies are working on education information systems that would integrate 
data on a broad range of transactions such as daily school attendance, grades and even library 
book checkouts, with the ultimate goal of automatically recording each interaction a student has 
with the school and student’s assessment and program participation data. 

Social structures and time set aside for analyzing and interpreting data. The most 
sophisticated data warehouse in the world will have no effect on instruction if no one has — or 
takes — the time to look at the data, reflect on them, and draw inferences for instructional 
planning. Given that time is one of the most basic resources in any organization, there need to be 
strong expectations that educators will take the time to examine data and use those data to guide 
improvements in their programs and practices. Teacher survey data show that setting time aside 
for such activities is not business as usual in American schools (U.S. Department of Education 
2008b). Case studies of schools that are active in data-informed decision making suggest that 
organizational structures that include time set aside for reviewing and discussing data in small 
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groups greatly increase the likelihood that the examination of data will be conducted and will 
lead to well-informed decisions (Choppin 2002; Cromey 2000). 

Professional development and technical support for data interpretation. Teacher 
training generally has not included data analysis skills or data-informed decision-making 
processes in the past (Choppin 2002). Few administrators have this kind of training either 
(Herman and Gribbons 2001). Moreover, the measurement issues affecting the interpretation of 
assessment data — and certainly the comparison of data across years, schools or different student 
subgroups — are complicated. Data misinterpretation is also a real concern (Confrey and Makar 
2005). For this reason, districts and schools are devoting increasing amounts of professional 
development time to the topic of data-informed decision making. Many argue that the practice of 
bringing teachers together to examine data on their students and relate those data to their 
practices is a valuable form of professional development in its own right (Feldman and Tung 
2001). Some districts are using Enhancing Education Through Technology (EETT) professional 
development funds to underwrite these activities. In addition, some districts that have been active 
in this area provide data “coaches” or other means for accessing technical expertise to school 
teams engaged in looking at data. 

Tools for acting on data. The examination of data is not an end in itself but rather a means 
to improve decisions about instructional programs, placements and methods. Once data have 
been analyzed to reveal weaknesses in certain parts of the education program or to identify 
students who have not attained the expected level of proficiency, educators need to reflect on the 
aspects of their processes that may contribute to less-than-desired outcomes and to generate 
options for addressing the identified weaknesses. Some of the data-informed decision-making 
systems incorporate resources that teachers can use in planning what to do differently. These 
resources are typically organized around state content standards and may include lesson plans, 
instructional materials or descriptions of best practices (Palaich, Good and van der Ploeg 2004). 
Resources for differentiated instruction can help teachers adapt their instructional approach to 
students with differing strengths and weaknesses. 

Data Sources for the Interim Report 

bindings from this report are drawn from both survey and case study data. The survey data 
include two data sets from the U.S. Department of Education’s National Educational Technology 
Trends Study (NETTS). The primary data set for this report consists of responses of a random 
sample of K-12 teachers to a survey administered to 2,509 teachers in spring 2007. The teachers 
were clustered in schools sampled from districts participating in a NETTS district survey. The 
NETTS district survey was sent to a nationally representative sample of 1,039 districts also 
surveyed in spring 2007. Both teachers and districts were asked to report on activities during the 
2006-07 school year.^ 

Eor the case studies, the site selection process began with identifying districts that have 
been active in using student data to inform instruction. The research team then worked with the 
selected districts to identify appropriate schools. By focusing fieldwork on districts where many 



^ The response rate for the teacher survey was 86 percent, and the response rate for the district survey was 94 
percent. Additional information about the NETTS survey samples can be found in Appendix A. 
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teachers could be expected to be actively looking at student data, the study team increased the 
likelihood of seeing effects of data use on practice, compared with a sample of schools drawn at 
random. 

District sites were identified through three methods: 

• Nominations by TWG members and other leaders in educational technology, 
researchers, vendors and staff of professional associations such as The Consortium for 
School Networking and State Educational Technology Directors Association, 
supplemented by a search of conventionally published and Web-published literature on 
data-informed decision making, to identify schools and districts with reputations as 
leaders in this area 

• Recommendations by ED staff members 

• Phone interviews with relevant staff members in nominated districts 

The final set of 10 districts for the first round of site visits in 2006-07 comprised three 
districts using their own locally developed systems and seven using commercial systems (each 
district using a different commercial system). They represented nine different states and 
comprised six large and four medium-size districts, with student enrollments ranging from fewer 
than 6,000 to more than 164,000 students.^ Student poverty levels ranged from 13 percent to 70 
percent of student enrollment, and the percentage of minority students ranged from 12 percent to 
87 percent. 

Eor each of the 10 districts selected for study, the study team contacted the district 
representative by phone to gain recommendations for three case study schools, based on the 
following criteria: 

• One elementary or middle school that the district considers high in its data use practices 

• One school that has shown improvement in its use of data to guide instruction 
(emerging) 

• One school that is typical of the district with respect to use of data systems 



Researchers asked the district to recommend, to the extent possible, three schools serving 
demographically similar students at the same grade level (either elementary or middle school). 



* Conclusions based on the case study districts should not be extended to all districts because they are not a 
nationally representative sample, but they do provide more in-depth information about data use practices that can 
be brought to bear in interpreting data gathered through national surveys. In particular, the generalizability of data 
from the case study sample to small districts (enrollments of fewer than 5,000 students) is limited; none of the 
districts visited in 2006-07 was small because small districts could not satisfy the requirement for three schools 
from the same grade span (i.e., three elementary or three middle schools). These smaller districts serve 
approximately 32 percent of the student population in public elementary and secondary districts in the nation as 
reported in the National Center for Education Statistics Common Core of Data (Hoffman 2007). District size can 
have a significant effect on the resources that school districts can apply to technology and professional 
development. 



7 




Within each school, the principal, an instructional or data coach (if applicable), and six teachers 
were interviewed. 

One of the 10 selected districts (large with a locally developed system) was unable to 
accommodate a site visit until the 2007-08 school year. The demographic and achievement data 
for the nine districts in the 2006-07 site visit sample are shown in Exhibit 1-2. For each district 
visited, respondents included key staff members involved in the district’s data-informed decision 
making activities (e.g., chief information officers, directors of curriculum and instruction, 
directors of research and evaluation, directors of accountability, directors of professional 
development). 



Exhibit 1-2 



2006-07 Case Study Districts 



District 

and 

Schools 


Demographics 


Percentage 
Attaining 2006 
Proficiency Levels 


State Average 
Proficiency Levels 


District 1 


Student enrollment = 132,482 (very 


Reading (Gr. 8) = 48% 


Reading (Gr. 8) = 41% 


3 middle 
schools 


large) 

Percentage minority = 74% 
Percentage poverty = 62% 
No. of schools = 211 


Math (Gr. 8) = 11% 


Math (Gr. 8) = 26% 


District 2 


Student enrollment = 164,295 (very 


Reading (Gr. 4) = 87% 


Reading (Gr. 4) = 87% 


3 elementary 
schools 


large) 

Percentage minority = 50% 
Percentage poverty = 20% 
No. of schools = 238 


Math (Gr. 4) = 83% 


Math (Gr. 4) = 81% 


District 3 


Student enrollment = 137,798 (very 


Reading (Gr. 4) = 90% 


Reading (Gr. 4) = 86% 


3 elementary 
schools 


large) 

Percentage minority = 59% 
Percentage poverty = 39% 
No. of schools = 199 


Math (Gr. 4) = 89% 


Math (Gr. 4) = 86% 


District 4 


Student enrollment = 39,213 (large) 


Reading (Gr. 4) = 84% 


Reading (Gr. 4) = 84% 




Percentage minority = 82% 


Math (Gr. 4) = 86% 


Math (Gr. 4) = 88% 


2 elementary 


Percentage poverty = 64% 


Reading (Gr. 8) = 88% 


Reading (Gr. 8) = 89% 


schools & 1 
middle school 


No. of schools = 61 


Math (Gr. 8) = 86% 


Math (Gr. 8) = 71% 


District 5 


Student enrollment = 26,229 (large) 


Reading (Gr. 4) = 39% 


Reading (Gr. 4) = 63% 


3 elementary 
schools 


Percentage minority = 87% 
Percentage poverty = 70% 
No. of schools = 55 


Math (Gr. 4) = 32% 


Math (Gr. 4) = 54% 


District 6 


Student enrollment = 5,599 (med.) 


Reading (Gr. 4) = 51% 


Reading (Gr. 4) = 56% 


3 elementary 
schools 


Percentage minority = 64% 
Percentage poverty = 43% 
No. of schools = 14 


Math (Gr. 4) = 43% 


Math (Gr. 4) = 48% 


District 7 


Student enrollment = 9,685 (med.) 


Reading (Gr. 4) = 82% 


Reading (Gr. 4) = 88% 


3 elementary 
schools 


Percentage minority = 58% 
Percentage poverty = 61% 
No. of schools = 19 


Math (Gr. 4) = 63% 


Math (Gr. 4) = 81% 






Exhibit 1-2 (continued) 
2006-07 Case Study Districts 



District 8 

3 elementary 
schools 


Student enrollment = 10,780 (med.) 
Percentage minority = 71% 
Percentage poverty = 62% 

No. of schools = 22 


Reading (Gr. 4) = 39% 
Math (Gr. 4) = 50% 


Reading (Gr. 4) = 64% 
Math (Gr. 4) = 71% 


District 9 

3 middle 
schools 


Student enrollment = 22,174 (med.) 
Percentage minority = 12% 
Percentage poverty = 13% 

No. of schools = 29 


Reading (Gr. 8) = 63% 
Math (Gr. 8) = 60% 


Reading (Gr. 8) = 43% 
Math (Gr. 8) = 42% 



Note; Numbers have been used to label districts for confidentiality reasons. The same number for a district is used 
throughout the report to support comparisons of data. Achievement data refer to the percentages of students who have 
scored proficient or above in reading and math in comparison with state standards based on state Web-site data. The 
number of district schools excludes early childhood centers and charter schools. 



Teachers to be interviewed were nested within the purposive sample of schools. Even 
though schools were selected because they used student data systems in instructional decision 
making, variability was expected across teachers with respect to how they were using data for 
instructional decision making. (Such variability is typical for educational reforms and was found 
for data-informed decision making in earlier school case studies (Marsh, Pane and Hamilton 
2006)). Therefore, project staff members requested that the principal of each case study school 
nominate three active practitioners of data-informed decision making and three teachers who 
represented average use. In this way, the study expected to capture “best practices” within the 
school but still maintain a realistic perspective with regard to the pervasiveness of those 
practices.^ 

An additional round of case studies will be conducted in school year 2007-08. A second 
sample of districts has been drawn from the pool of districts that remained after the initial 
selection of 10 districts, supplemented by additional districts identified as being involved in data- 
informed decision making activities. These districts included Broad prize winners and nominees 
as well as selected districts participating in focus groups at the U.S. Department of Education to 
discuss issues related to education technology (i.e., districts using data to support instructional 
decision making). The 2007-08 case study sample also includes a small district (for which an 
exception was made to the requirement of having three schools serving the same grade levels). 

Contents of This Report 

This interim report focuses primarily on describing the types of data available to school 
staff members, how school staff members use electronic data systems, school practices with 
respect to data-informed decision making, and the supports and challenges for school use of 



^ It is quite possible that the teachers whom principals described as typical in terms of data use were in fact better 
than average for the school. When this potential bias is considered along with the fact that the study conducted 
case studies in districts that were considered leaders in the instructional use of data, the reader should be aware 
that the teachers’ understanding of data described herein is likely to be better than that of average teachers. 

The districts participating in the focus groups were identified through expert nominations and purposively 
sampled to obtain a balance by size (small, medium, large) and geographic region. 
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student data in planning and implementing instruction. In addition, Chapter 4 describes findings 
from the presentation of data scenarios to teachers who were asked a series of questions related 
to how they could understand and use hypothetical data sets. The chapters of the report are 
organized around these issues, with survey and case study data on the same issue presented 
together. Chapter 5 summarizes the study’s early findings. 
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2. School Data Practices 



Most states and districts have relied on cohort data to document academic achievement. This 
type of data is often used to present results of student performance on annual assessments and to 
compare schools and districts. What cohort data do not provide is information on the progress of 
individual students over time or the capability to disentangle changes in achievement from 
changes in the composition of the student cohort. 

Recently, considerable effort has been put into building data systems at the state level that 
are capable of tracking performance of students over time. For example, the Institute of Education 
Sciences in the U.S. Department of Education awarded grants of $52.8 mi llion in 2006 and $62.2 
million in 2007 to state education agencies (SEAs) for the design and implementation of 
statewide longitudinal data systems. The Data Quality Campaign, a national collaborative effort 
to encourage and support the implementation of state longitudinal data systems to improve 
student achievement, has sponsored an annual survey of states to assess how many of the 10 
“essential elements” of a longitudinal data system each state currently has (Data Quality 
Campaign 2007).“ 

The data collected in this study suggest that the extensive state-level activity around 
longitudinal data systems has not yet had a major effect on local use of student data. Much of the 
data available to educators comes from local data systems. In the 2006-07 NETTS teacher 
survey, among teachers reporting access to an electronic data system that provided them with 
student data, the vast majority (79 percent) reported that the data system was provided by their 
district. Just a third of these teachers (33 percent) reported that the data system was by their 
school, and just 17 percent said that it was provided by their state (respondents could indicate 
more than one source). Teachers in the nine NTA case study districts also relied on their 
districts’ electronic information systems as the primary source for student data. Case study 
districts were located in states with state data systems that represented a broad continuum in terms 
of implementation of a longitudinal data system: the systems within these nine states had between 
three and nine of the Data Quality Campaign’s 10 essential elements of a longitudinal data system 
(a summary is provided in Exhibit A-2 in Appendix A). 

District Data Systems 

Proponents of data-informed decision making in education offer the vision of an integrated 
system of systems, combining longitudinal information on each individual student’s school 
history, data needed for accountability as well as finance, personnel, food service, and even 
transportation data (U.S. Department of Education 2004; Wayman 2005). This vision is valuable 
in setting a direction for policy-makers, but even in districts nominated as leaders in the use of 
student data systems, it is still not a reality. Typically, the information needed for educational 
decision making is spread across multiple systems without mechanisms for regular transport of 



" In 2007, among the 50 states and the District of Columbia, three had one to three elements, 1 1 had four to five 
elements, 21 had six to seven elements, 12 had eight to nine elements, and four had all 10 elements. 

Even though teachers using a data system are most likely to be using a system provided by their district, a good 
portion of the data in the district’s system may have come from the state system. 
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information from one system to another. When asked what system provided student data that 
sehools use for decision making, staff members at the nine districts in the spring 2007 case study 
sample all mentioned two or more separate systems. A summary of the types of data systems used 
in the case study districts is included in Exhibit A-3 in Appendix A. 

This state of affairs is a natural outgrowth of the multiple purposes for which student 
information is collected and the fact that systems have tended to be developed (and marketed) 
over time around a range of specific needs. The typical district has a data system for tracking 
required special education services, for example, which is separate from the system that maintains 
the longitudinal enrollment histories of students generally. Some systems are geared to capturing 
daily transactions such as attendance and grades while others are designed for access to 
longitudinal records (Wayman 2005). Although it is not necessary to have all types of data in a 
single system, lack of interoperability between systems creates inefficiencies in data input, and 
differences in system interfaces increase training requirements for district and school staff. 

Wayman (2005) defines four kinds of student data systems in widespread use: 

1. Student information systems provide real-time access to student data such as attendance, 
demographics, test scores, grades and schedules. 

2. Data warehouses are electronic data collection and storage systems that provide access 
to current and historical data on students, personnel, finance, etc. 

3. Instructional/curriculum management systems provide a unifying framework to support 
access to curriculum and instructional resources such as planning tools, model lesson 
plans, creation of benchmark assessments, linkage to state content or performance 
standards, communication and collaboration tools (e.g., threaded discussion forums). 

4. Assessment systems support rapid organization and analysis of benchmark assessment 
data. 

The most frequent types of electronic data maintained by districts — based on all districts 
responding to the 2006-07 NETTS district survey — were student attendance (96 percent), grades 
(93 percent), demographics (92 percent), special education information (91 percent) and course 
enrollment histories (88 percent) — basic elements of a student information system. Student test 
scores on statewide assessments were maintained in electronic form by 86 percent of districts. 
Eewer districts stored electronic data on teacher qualifications (62 percent) and on participation of 
students in particular educational programs such as those using an innovative classroom 
curriculum (77 percent).*"^ 

The nine districts in the spring 2007 case study sample were similar to the national sample. 
They all had student information systems, but, as noted above, the data typically were captured in 
several different systems with some limitations in interoperability. Data warehouses with 



National estimates on the types of student data systems in districts and how long they have been available will be 
provided through the 2007-08 NTA district survey (Questions 3 and 4). 

Information on the types of data in district information systems will be updated through the 2007-08 NTA district 
survey (Question 5). 
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longitudinal data were available in about half of the case study districts, but curriculum 
management systems were used by just two. 

One theme that emerged from the case studies was the extent to which multiple systems 
were used to house education data, without necessarily having any efficient procedures (or 
sometimes without any capability at all) for connecting data from different systems. One district 
described the use of four different data systems: one for state and district assessment scores, one 
for student demographic and enrollment information, one for discipline reports, and one for 
special education. Across case study districts, it was typical to find separate systems for 
longitudinal data, teacher professional development data and special education data. None of the 
districts had an integrated data system and data warehouse of the kind described in the 2004 
National Technology Plan (U.S. Department of Education 2004, 44). 

Teacher Access to Student Data Systems 

Among the nationally representative sample of teachers taking the NETTS teacher survey, 
roughly three-quarters (74 percent) reported having access to an electronic student data system in 
school year 2006-07, up from 48 percent in the NETTS teacher survey conducted in 2005.^^ 
Although this trend in the survey data is encouraging, the school case studies provided an 
opportunity to get a deeper understanding of data system access issues. The question of data 
access is actually surprisingly complex. Does access mean that a teacher can get the data out of 
the system herself or just that there is a way that she can get data from the system if she asks 
someone for it? What if a teacher is allowed to see some kinds of data for his students but not 
other kinds, or what if he can see data for some of his students but not all? Does a teacher have 
access if she can get only standard reports and cannot frame any of her own data queries? 

Teachers in all of the case study districts had some form of access to student data, but 
teachers could get assessment data for their students out of the data system themselves in only 
four of the nine districts. Many districts consider having relevant data extracted for teachers to 
be a more efficient, reliable practice than trying to train teachers to extract data for themselves. 
Teachers could extract some but not all assessment data for their students in an additional two 
districts. In three districts, teachers received data reports from the district office but could not 

17 

extract student data on their own. 

Although case study districts were selected because of their activity in promoting data 
systems, teachers in the selected schools reported little firsthand use of data systems. It was rare 
for case study teachers to describe accessing student assessment data from a system themselves 
for any purpose other than analysis of their students’ responses to individual assessment items. In 



The 2005 NETTS teacher survey was administered to a nationally representative sample of K-12 teachers in fall 
and winter 2005. Teacher respondents were asked to report on activities in the 2004-05 school year. Teachers were 
sampled from 957 schools within districts selected for the NETTS district survey. These are the same districts that 
participated in the 2007 NETTS district survey (see U.S. Department of Education 2008a for additional 
information on these surveys). 

Among teachers sampled from six case study districts who responded to the NETTS teacher survey, 80 percent 
reported having access to an electronic data student data system in 2006-07 compared with 74 percent of teachers 
in non-case-study districts. 

Information on the access that school staff have to student data systems will be updated through the 2007-08 NTA 
district survey (Questions 9-11). 
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two-thirds of the districts, there were one or more staff members at the school site who would 
produce data reports for teachers. In a number of cases, this practice was a deliberate decision on 
the part of the district to provide this resource, either because teachers were viewed as already 
overburdened or because there were doubts about teachers’ ability to use the system or understand 
the data by themselves. Even a district that began by purchasing a laptop for every teacher to use 
for accessing the data system concluded that teachers would not do so with any frequency on their 
own and ended up creating assessment and instructional coach positions to help teachers connect 
data to instruction. Staff members in one case study district made it clear that they did not think 
that school access to student data systems would be a good practice. District staff members 
expressed concern that principals might use data from the system inappropriately to evaluate 
teachers. 

Types of Data Available to Teachers 

Among the 74 percent of NETTS teacher survey respondents who said that they had access 
to a student data system, the most common types of data available to teachers were attendance 
data (74 percent) and student grade data (67 percent) — the same types of data that districts 
reported frequently having in their electronic data systems. Teachers also reported access to data 
on their current students: 77 percent said that they could get access to standardized test scores 
(spring and fall 2006) for the students they were teaching currently, and over half (55 percent) had 
performance data on benchmark or diagnostic tests taken by their current students. Eess than half 
(46 percent) of teachers with access to a student data system reported having access to data for 
students they had taught the preceding year. A much smaller percentage of teachers (29 percent) 
reported being able to access multiple years of test scores for individual students. Only 8 percent 
said that they could get information on any supplemental services their students were receiving. 
Exhibit 2-1 summarizes these survey findings. 

Another perspective on teachers’ access to data from a student data system can be gained by 
examining responses of district personnel completing the NETTS district survey. Eess than half of 
the districts (46 percent) responding to the survey reported that their teachers are provided with all 
or most of the data available for their students. Eorty-four percent indicated that they provide 
teachers with only a limited set of data on their students. Another 9 percent of districts reported 
that they do not make any data available to teachers on the students in their classrooms. 
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Exhibit 2-1 

Data Available to Teachers in 2006-07 



Types of Data Available 


Percentage of 
Teachers With 
System Access 


Percentage of All 
Teachers 


Attendance data 


74 


55 


Student grade data 


67 


50 


Performance on benchmark or diagnostic tests 
taken by your current students 


55 


41 


Spring 2006 standardized test scores for the 
students you taught last year (2005-06) 


46 


34 


Spring 2006 standardized test scores for your 
current students 


44 


33 


Fall 2006 standardized test scores for your 
current students 


33 


25 


Multiple years of standardized test scores for 
individual students 


29 


11 


Course enrollment histories for students 


22 


16 


Students’ prior school(s) attended 


16 


12 


Students’ participation in supplementary 
education programs (e.g., tutoring) 


8 


6 



Exhibit reads: Among teachers who had access to an electronic student data system in 2006-07, 74 
percent reported that they had access to student attendance data through the data system, which 
represents 55 percent of all teachers (with or without access to a data system). 

Source: 2006-07 NETTS teacher survey. 



The purposes for which teachers reported using data from student data systems are shown in 
Exhibit 2-2. The most common purposes reported by teacher survey respondents were informing 
parents about student progress (68 percent), tracking individual student scores (65 percent), and 
estimating whether students are making adequate progress (64 percent). Additional purposes cited 
by a majority of teachers with data system access were tracking other measures of student 
progress (59 percent) and identifying skill gaps for individual students (55 percent). (Unless 
otherwise indicated, all percentages reported for teacher survey respondents in the remainder of 
this report are for the 74 percent of teachers who reported that they had access to an electronic 
student data system in 2006-07.) 



Teachers responded that they used an electronic student data system for a specified activity for one of a set of 
specified frequencies during the 2006-07 school year (i.e., a few times, once or twice a month, or once a week or 
more). 
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Exhibit 2-2 

Teachers’ Use of Student Data 




0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 

■ Once a w eek or more □ Once or tw ice a month □ A few times □ Never □ Data not in system I Percontagc of tcachers 



Exhibit reads: Among teachers who had access to an electronic student data system in 2006-07, 1 1 percent 
responded that they used data to inform parents about student progress once a week, 20 percent once or twice a 
month, 37 percent a few times, and 19 percent never; 13 percent indicated the system did not provide data for this 
purpose. 

Note: Totals may not add to 100 percent because of rounding. 

Source: 2006-07 NETTS teacher survey. 

The extent to which teachers conduct these data use activities is likely to be influenced by 
teachers’ perceptions of system capabilities. For example, about a third of teachers reported that 
the system they had access to did not provide data to inform curriculum changes (32 percent) or to 
evaluate promising classroom practices (31 percent). A potential source of ambiguity in teachers’ 
data use reports is teachers’ varying interpretations of their roles (as well as the role of data) in 
functions such as curriculum change and evaluation of practices; these functions may be seen as 
district responsibilities. In addition, districts may have policies that run counter to teachers’ use of 
data for some kinds of decision making. For example, 63 percent of teachers reported knowing 
how to use student data to refine what and how they teach, but 60 percent also reported that 
regardless of what the data tell them, they need to keep up with state- and district-mandated 
pacing plans. 

In the case study schools, teachers commonly described the use of data for placing students 
in different classes or programs and as a basis for grouping students within classes for 
differentiated instruction. Teachers reported that the use of assessment data for this purpose had 
increased their confidence in their decisions. As one reported, “Before, we had nothing to back up 
our professional judgment. Now we can pinpoint where a student has a problem. It has increased 
our workload, but it has made us better at diagnosing.” Teachers also reported using data (from 
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benchmark assessments) to decide whether to reteach. It was less common for teachers to report 
the use of data as a stimulus for reflection on their own teaching approach, but instances of this 
use were observed in four districts. For example, at one school, teachers use a team approach to 
problem solving. Teachers work together in their grade-level teams to analyze and talk about data. 
If teachers find that their classes are performing at low levels on a particular standard compared 
with those of some of their colleagues, they work with other teachers to get ideas for ways to 
better teach that standard. 

A use of data reported in case study schools that was not included in the NETTS survey 
item was for school improvement planning. Interviewees described cases in multiple districts in 
which the state mandated annual improvement planning, and either the district or the principal 
brought in school data as part of this process. 

Social Context for Data Use 

Among NETTS teacher survey respondents, the most commonly reported context for using 
a student data system was on one’s own (78 percent), followed closely by data use with 
colleagues in a department or grade (71 percent). Eifty-nine percent of teachers said that they had 
used data as part of a district-run activity for staff members from their school. Just 28 percent had 
done so as part of a district activity for teachers from multiple schools, suggesting that districts 
view the school as the appropriate unit for professional development in this area. 

In each of the case study districts, the study team found schools where teachers looked at 
data on their own and in groups. The most frequently cited groupings for data use were the grade- 
level team, sometimes facilitated by a coach, and all-staff faculty meetings. Other types of 
groupings in which teachers used data included cluster team meetings, curriculum groups 
(department teams in middle schools), schoolwide special-purpose teams, and cross-grade teams. 
In one district, teachers at the high-data-use school are actively involved in data discussions 
during weekly grade-level meetings, where they examine data collectively and use the data to 
help them form cross-class groupings for reading. Additionally, there are cross-grade meetings 
during which teachers discuss student scores, skills that students are struggling with, and how 
teachers should work collaboratively across grades. These cross-grade meetings were initiated 
when data identified low math scores as a schoolwide problem. The school has also set up its own 
server where teachers can share instructional materials, assessment data and student work and 
where primary-grade teachers can store reading assessment data that are not kept on the district’s 
electronic data system. 

Teachers at a school in another district are encouraged to use data collaboratively and on an 
ongoing basis. The assistant principal and reading specialist retrieve data monthly from the 
district data system for each tested grade level and put it into a spreadsheet so school staff 
members can examine student performance by teacher, by grade, by subgroup or by educational 
program. These spreadsheets are updated after every test window in reading and math so teachers 
can bring the reports to their team meetings to discuss student grouping. Twice a week, the school 
holds grade-level team meetings where teachers bring their own data and discuss which students 
need academic support, which students can graduate from academic support, which students need 
coaching in an accelerated class, and which students need to be moved out of the accelerated 
class. Teachers find it very important to learn how well students in other classes are doing 
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because this information is useful when they plan their lessons collaboratively. In addition, grade- 
level teams meet with the principal three times a year to discuss data and instructional 
strategies — how well students have performed overall, how the academic support works in the 
classroom, what new goals they have set, and what they need to achieve their objectives. The 
principal at this school believes that teachers should know how to access and use data. To support 
this process over the past three years, the principal has provided all teachers with school-based 
training on how to use the data system and Excel. 

Barriers to Use of Data Systems to Inform Instruction 

Mandinach et al. (2005) have identified a set of factors that contribute to teachers’ use of 
technology tools for data-informed decision making: easy access; short feedback loops (between 
taking assessments and receiving results); information comprehensibility; flexibility (allowing 
data to be examined in different ways); and alignment with standards, instructional goals and 
classroom practice. Teachers participating in the NETTS survey and those in case study schools 
cited shortcomings in these areas. 

A sense of what issues are getting in the way of teachers’ use of data can be gleaned from 
teachers’ ratings of attitude -related statements on the NETTS teacher survey (displayed in Exhibit 
2-3). The biggest concern expressed by surveyed teachers was with their ability to use a system to 
form data queries. Whether teachers feel hindered because they do not know how to frame queries 
or because the system they have does not support user queries, only 33 percent of teachers said 
that they could form data queries to get pertinent data from the system. The other most commonly 
cited issues were the limited utility of the kind of information available in the student data 
systems for deciding what and how to teach (cited as a barrier by 29 percent); difficulty finding 
the information they are looking for on the system (24 percent); and difficulty in using the system 
(20 percent). Among school staff members who were interviewed as part of the case study 
research, the three most commonly cited barriers to school use of data were (a) lack of training in 
how to use the data system or to derive instructional implications from it, (b) the lack of time to 
engage in data exploration and reflection, and (c) the weakness of the available data. State 
assessment data were criticized as coming too late in the year to be very useful in some districts 
and as being too gross a level to provide guidance on anything more specific than major topics or 
proficiency categories. District benchmark assessment data are collected more frequently and 
made available while teachers are still responsible for the examinees, but teachers in multiple case 
study districts questioned the quality of their districts’ tests and their alignment with the 
curriculum. 
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Exhibit 2-3 

Teacher Experiences Using Electronic Student Data Systems in 2006-07 



Statement 


Percentage of 
Teachers Who Agreed 
or Strongly Agreed 


1 am capable of forming data queries (asking specific 
questions and getting the pertinent data from the 
system) 


33 


The student data available are not really that helpful in 
deciding what or how to teach 


29 


1 have trouble finding the information 1 want on the 
electronic student data system 


24 


The electronic student data system available to me is 
hard to use 


20 



Exhibit reads: Among teachers with access to an electronic student data system, 
33 percent felt capable of forming data queries. 

Source: 2006-07 NETTS teacher survey. 



District staff members who were interviewed as part of the case study site visits provided 
their opinions concerning the barriers to data system use and data-informed decision making. 
District staff members voiced concerns similar to those articulated by teachers. Staff members at 
seven of the nine case study districts cited limitations in their data systems, including lack of 
system operability, cumbersome processes for generating custom reports, and lack of technology 
at the school level that would support teachers’ access to and use of the data system. Staff 
members at seven districts cited limitations in the nature or timing of the data in the systems. As 
with teachers, the major complaints from district staff members were that the standardized test 
data do not provide enough information about students’ specific skills and weaknesses to support 
instructional planning, that the system does not have data from formative (mid-year) assessments, 
and that state test data come too late (typically in the fall for the prior spring’s examinations) to 
support school planning. Equally common among district respondents was concern that school 
staff members (both principals and teachers) lack the skills needed to use the data system and to 
analyze data effectively (seven districts). Other common district concerns (each expressed in five 
districts) were lack of buy-in from school (and some district) staff members, lack of assessments 
well aligned with curriculum standards or for primary grades, and lack of school-level staff 
members who could help teachers learn to access and use data from the system. Staff members at 
four districts expressed concern with resource requirements; administering assessments, 
implementing a data system, entering data and discussing data were all cited as activities 
consuming large amounts of staff time.^^ 

An issue underlying many of the tensions around local use of data systems is the tradeoff 
between standardization and customization. The more powerful software systems for storing, 
manipulating and reporting student data require major expenditures, not only for software 



The NTA district survey will provide national estimates on current barriers to expanding data-informed decision- 
making practices (Question 25) and areas where districts and schools need more support for data system use and 
data-driven decision making (Question 24). 
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purchase but also for the hardware needed to host it and to provide school-level access, for staff 
and teacher training around system use, and so on. On the one hand, district investments in such 
systems typically are predicated on the assumption of a uniform set of reporting and data use 
practices across schools. Teachers and schools, on the other hand, often feel that the kinds of 
information available in a district system do not address the questions and decisions most 
important to them. School staff members often work with less formal, home-grown data sets such 
as student performance on school-developed final examinations that are kept in electronic 
spreadsheets. Districts like to see districtwide benchmark assessment data in the districtwide 
system, but (as noted above) teachers often expressed misgivings about the quality of the district 
assessments and therefore the usefulness of the data. 

There may be strategies for resolving this tension. In one case study district, implementation 
of their student information system made district staff members aware of teachers’ need for 
formative assessments and of the lack of districtwide tools for measuring end-of-course 
achievement. The data system prompted a district effort to have the curriculum specialist in each 
content area work with teacher representatives from each school to craft end-of-course 
assessments that all schools would use. Implementation of these assessments and a new data 
system designed to support teacher access to the resulting data is expected in school year 2007- 
08. 



Districts do not necessarily want teachers to have direct access to student data. Although 
those advocating for educators’ use of data for decision making describe active involvement by 
teachers in data-based inquiry (Brunner et al. 2005), school districts are far from unanimous in 
promoting this model. Even among the nine districts known for their promotion of data-informed 
decision making that constituted the case study sample, only four had direct teacher access to the 
data system as an active goal. An additional four districts viewed teacher access as a goal for the 
longer term, after the system is made more user friendly or after more teacher training. One 
district said that it had decided to limit teachers’ access to the data system out of concern that data 
would be used inappropriately. Even principals in that district had access only to standard reports. 

Having school-based staff members who can act as bridges between teachers and the data 
system appears to be an important facilitator. Six of the nine case study districts have school- 
based positions to support teachers’ use of student data. A seventh district instituted district- level 
coach positions to help schools work with data. These positions are held by various types of staff 
members in different districts. The potential downside to this practice, as noted by Boudett and 
Moody (2005), is the risk that the generation and examination of data will come to be regarded as 
the responsibility of a single person within the school, without involving the school staff as a 
whole. The benefit of having this position is that this staff person can promote data use among 
teachers, provide access to data that teachers otherwise would not see, and model data analysis 
and reflection. Eor the most part, individuals in these positions in schools in the case study sample 
were involved in working with teachers on instructional issues. In several districts, an 
instructional coach or advisor led the school-level data use activities, sometimes in connection 
with a particular initiative such as Reading Eirst. In some cases where there was no school-based 
position to support data use, principals or vice principals were trained on data system use and then 
expected to lead their teachers in these activities. 
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Although teacher access to student data systems has increased over time, the types of data 
available to teachers are not necessarily useful for instructional decision making. Survey and case 
study data suggest that many barriers still exist to the effective use of data by teachers, including 
the need for more user-friendly data systems, coupled with the need for additional training on 
how to use these systems and how to analyze data. In addition, many of the supports required for 
data-informed decision making are lacking. The district and school supports that are available to 
teachers are described in greater detail in the next chapter. 
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3. District and School Supports for Data-Informed Decision Making 

The idealized model of data-informed decision making supports that are described in 
Chapter 1 provides a framework for examining the practices of districts and schools with respect 
to data use. This chapter explores the issues of data system integration, district and school 
leadership for data use, tools for generating and organizing data, and tools for acting on data that 
were found in actual practice in the case study districts and schools. 

Data System Integration 

Although schools are being exhorted to base their decisions on data, the systems available 
to them were not necessarily designed with their needs foremost in mind. Case study data 
suggest that most student data systems were built to address uses other than school-level 
instructional decision making. Moreover, the multiplicity of systems used by local educational 
agencies works against the integration of different kinds of information (e.g., discipline and 
attendance data, program placement information, district benchmark and state test scores) and 
requires school personnel to go through the ramp-up time to learn to use multiple systems. 

All the districts in the case study sample had student data systems, but none had a fully 
integrated set of data systems. One district, for example, had an instructional management 
system that included the district curriculum standards, curriculum guides, instructional resources, 
all student assessment data and student demographics. A data warehouse stored the district’s 
longitudinal data, and an online professional development data system was used to develop 
teacher professional development plans. Another district used three data systems. The student 
information system provided historical test information for each student, including past state and 
district tests (but not current scores). A separate Web-based transactional system was used by 
schools to transmit to the district information such as attendance, school lunch data and scores on 
district benchmark tests. This system included a grade book that was available, but not required, 
for teacher use. A separate system housed special education data.^° 

In general, it was quite common not only for districts to have separate systems for 
transaction capture (attendance and grades) and assessment data but also for these systems to be 
detached from instructional resources geared to standards. Hence, although the two NETTS 
teacher surveys suggest that teacher access to data systems increased dramatically between 2005 
and 2007, the systems themselves do not necessarily support integrated planning and self- 
evaluation. 

District and Schooi Leadership for Data Use 

Districts were nominated for case study on the basis of their leadership in the use of data 
for decision making, and not surprisingly, evidence of leadership in this area was found in all of 
them. Case study districts demonstrated their support and leadership for schools’ use of data 
through a number of actions: 

• Making data-informed decision making a priority and purchasing or developing a 
system that supported these activities 



The district was in the process of developing a data warehouse to link data from all these systems. 
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• Considering ability to use data as a criterion for teacher hiring (One district had 
developed interview questions on the use of data for instructional decision making and 
used these when hiring teachers. Another had developed this kind of question for use 
when hiring district staff members.) 

• Providing training and support positions for system implementation (and tracking 
usage). 

• Districtwide data-informed decision making activities in addition to professional 
development (An example of this kind of activity is provided in Exhibit 3-1.) 

• Using the data system for decision making at the district level (e.g., evaluating 
programs, principals, teachers). 



Exhibit 3-1 

A District and School Data Analysis Process 

Every year, schools in this medium-size district get together to conduct “data digs” where school 
leadership teams have the opportunity to review and discuss their state test data. The teams use 
a collaborative inquiry process called the Data-Driven Dialogue to analyze and interpret their 
data. The process, adapted from Bruce Wellman and Laura Upton (2004), can be used with any 
kind of data from any source. The process includes four steps: 

• Predict — activate and engage interest in the data, access prior learning, name frames of 
reference, and establish common ground for dialogue. 

• Explore — Interact with the data, look for patterns and trends, identify data facts and 
surprises, make observations without inference, identify questions raised by the data, and 
develop problem statements. 

• Explain — generate theories of causation, stay open to multiple possibilities, deepen 
thinking and identify “root causes” rather than symptoms, make inferences about data, 
and identify the additional data that will validate the theories of causation. 

• Take Action — move from problems to solutions based on validated theories of causation, 
identify goals and specific related action steps, and identify data to be monitored to 
determine whether action steps lead to the solution. 

All schools were initially provided training on the process by the State Consortium for Data-Driven 
Dialogue — District 8 is the lead district. The consortium was established with support from a 
$3.5 million, three-year grant from the SEA with federal EETT funds. 

Information from the data digs is used to inform the development of the school’s improvement 
plan, which is a requirement of the state accreditation process. Schools are then encouraged to 
continue to review their data throughout the year by means of a monthly action plan that 
implements and monitors progress on goals in the school improvement plan. School leaders who 
have embraced the use of data-informed decision making use the same data inquiry cycle to 
develop their monthly action plans and to work with grade-level teams to monitor student 
progress and progress on school improvement goals. Principals reported that the data digs have 
helped them to structure conversations with their staff members about data. Teachers can look at 
student progression in a particular skill or content area as a way to monitor their own instruction. 



Any increase in entry-level requirements has the potential to screen out job candidates with otherwise desirable 
attributes. However, no evidence was identified, one way or another, specific to any entry-level requirements for 
teachers’ ability to use data. 
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Two districts were judged to have weaker leadership for data use than the other seven 
because one had a very compartmentalized approach to data use (there was little sharing across 
departments even though the district was an advocate of data use) and the other district was just 
beginning to address a more comprehensive approach to data use at the district level. Neither 
district is currently providing school-based staff members to support teachers’ use of data (both 
districts have faced declining resources that have forced support staff cutbacks). Although both 
districts are acquiring new data systems that will enhance data accessibility and analysis 
capabilities, currently, neither has a plan for how it will provide the necessary professional 
development and ongoing support to school staff members to use these new systems. 

At the school level, principals can show leadership with regard to data use by setting a 
personal example of data use, establishing expectations for data-informed decision making and 
working with teachers to interpret data. (An example of such a principal is provided in Exhibit 3- 
2.) Across case study schools, principals showed strong leadership in less than half of the 
schools. In two of nine districts, principals at all three schools set an example of data use. There 
was no clear pattern evident between principal leadership and school type (typical, emerging and 
high) in the case study sample, although principals who acted as data leaders were slightly more 
common in high-use and emerging schools than in typical schools. Principal leadership may 
appear less strongly related to data-informed decision-making activity in this study’s sample than 
in prior case studies because of the presence of other school leaders or coaches assigned to work 
with teachers on data use. 



Exhibit 3-2 

Principal as Data Leader 

The principal at a school identified for its exemplary use of data uses a broad range of data, 
including students’ report cards and grades. She believes that grades, like behavior and 
attendance, are a reflection of student engagement. She also looks at enrollment data (e.g., if 
students selected the school by choice) and reviews individual target students’ data with their 
teacher teams. The principal feels that teachers need data in their hands, and even though they 
have access to the district data system, she cleans and organizes school-level standardized 
testing data from the system for her teachers. Her philosophy is that it is much more important 
that teachers work with the data to understand it rather than work to organize it. She meets with 
teachers twice a month to talk about the instruction behind the test scores and provides ongoing 
data to support them. As the school year progresses, school staff members are encouraged to 
use a variety of data from individualized education programs (lEPs) and from in-house, district 
and specialized benchmark assessments to monitor students within their classes and 
departments. Teachers are expected to use these data to reflect on and discuss with their grade- 
level teams what is and what is not working instructionally. Teachers also base their initial 
referrals to have students assessed for special education on this review process. As a result, a 
culture of data-informed decision making has taken root at the school. 



Schools set expectations for data use when they set aside time during school hours to 
discuss data. All schools in the case study sample provided at least minimal opportunities to 
discuss data (e.g., at department meetings or during a common planning time available to 
teachers of a particular grade or subject). The amount of time available for these discussions 
varied widely across districts and was only loosely correlated with schoolwide data practices. 
One district provided opportunities to discuss data in all schools two to three times per week. In 
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another district, the high-use school used common preparation times and monthly professional 
development meetings to talk about data. This type of discussion occurred less frequently at 
departmental meetings in the emerging and typical schools in the same district, where principals 
often shared data only at the beginning of the school year. 

School leaders can also encourage data use by working with teachers to help interpret their 
data. In some cases, the principal filled this role; in other cases, specialists and lead teachers did 
so. As research on school leadership has shown (e.g., Cambum, Rowan and Taylor 2003), when 
there are subject leaders in the school, they can play a role equal to that of the principal in terms 
of influencing instruction. In some case study districts, designated instructional or data coaches 
took on — or at least shared with the principal — the role of data leader. When coaches were 
present in a school, teachers were more likely to work with the coaches than with their principals 
on data interpretation, but in some of these schools, principals supplemented teacher meetings 
with the coach with their own meetings where they led the data discussions with teachers. In one 
district that did not have school-based coaches (they had been lost to budget cuts), the principal 
in the school nominated by its district as “high” in terms of data use was more active in working 
with teachers to review their data and discuss instructional implications than were the principals 
in schools described by the district as “emerging” or “typical” in terms of data use. In general, 
however, school-level supports for data use were more similar within than across districts. 

Tools for Generating and Organizing Data 

Another way for districts to promote data-informed decision making is to provide tools for 
generating and organizing data. Data systems that provide student achievement data in a form 
that can be disaggregated by student category through standard or custom reports help reduce the 
burden of analyzing data for school staff members. Some systems can provide data not only from 

standard and formative assessments but also from daily transactions such as school attendance or 
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from receipt of special services such as tutoring. 

The case study districts represent a range of system capabilities with regard to providing 
formative assessment data, but all provided some form of reporting of student scores broken 
down by skill areas that schools could use for planning purposes. Four districts had data systems 
that incorporated student scores on interim assessments that were aligned with the state 
standardized tests, and a fifth district had some of its interim assessment data available through 
its data system. Two districts provided interim assessments that were not aligned with the state 
test, and another two districts were developing benchmark assessments that will be available in 
the data system at some future date. Even though all districts had systems that allowed analysis 
of student performance in specific skill areas, some district systems were more robust than 
others. For example, in one district, skill areas were sometimes covered by only one item on the 
assessment, making it easy for teachers to dismiss the assessment data. In another district, skill- 
level analysis of test data was possible for grades 3 and above, but not for primary-grade 
students. The systems in two districts provided standard reports by skill area, and they system in 
a third district allowed custom reports to be generated. 



The 2007-08 NTA district survey will provide prevalence estimates for tools for gathering data (Question 8). 
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Online assessments were less common in case study districts, with only two districts 
providing online assessments for any subject. Several other districts that did not provide online 
assessments did have other technology supports for assessment. Three had digital item banks that 
allowed schools to develop their own formative assessments, either in addition to or in place of 
district benchmark tests. Other districts used handheld computers, which allowed teachers to 
enter data as they administered tests and to generate reports instantly. In the three districts where 
use of handheld computers was a relatively common practice, handhelds were used for reading 
assessments in the early grades. 

Organizational Structures Supporting Data Use at the School Level 

Organizational structures that support data use at the school level can include (a) time set 
aside for teachers to review and discuss data in small groups, (b) designated support staff and (c) 
the adoption of procedures for discussing data. On the 2006-07 NETTS teacher survey, most 
teachers reported having positive perceptions of support for using and interpreting the data that 
they could access from data systems. A majority of teachers with access to a student data system 
(71 percent) agreed or agreed strongly that when they needed help making sense of the data in 
the system, they knew someone who could help, and 45 percent reported that someone else 
usually extracted the relevant data from the system for them. 

Of those teachers who indicated on the NETTS 2006-07 teacher survey that they had 
access to a student data system, 58 percent reported that they received support for using data to 
guide instructional decisions from professional development received at their school and 
56 percent reported receiving encouragement for using data from their school principal. A 
quarter of these teachers (25 percent) also reported receiving support from a consultant or mentor 
teacher skilled in data analysis. 

Working in small groups can also support teacher use of data. Of the teachers who 
indicated that they had access to a data system on the 2006-07 NETTS teacher survey, 70 
percent said that they were comfortable having their colleagues around when they examined 
performance data for their students, and 59 percent said that they knew how to work with 
colleagues in using student data to monitor progress and set goals. 

In many districts, teachers were not compensated for the time they spent working with data. 
Eess than a quarter of teachers (23 percent) had time available during the regular day for 
examining data, and a majority (59 percent) reported accessing the data system on their own 
time. Only 12 percent reported that they had other paid time set aside for examining student data 
and using data to guide decision making about practice. 

Case study districts provided stronger support for data-informed decision making than the 
typical level indicated in the NETTS teacher survey responses. As mentioned in the preceding 
chapter, a majority of case study districts had used school-based staff members to act as bridges 
between teachers and the data system. Six out of nine offered some sort of district-funded, 
school-based staff to support teachers’ data use, but these staff varied in number, amount of time 
available and expertise. One district had just two district- level coaches to serve all 10 of its 



The 2007-08 NTA district survey will provide prevalence estimates for district supports for school use of data 
systems to inform instruction (Question 20). 
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elementary schools, while others had a coach assigned to each school. School-based coaches 
went by different titles such as data facilitator, technology specialist, system certified trainer or 
staff development teacher and often had other responsibilities such as instruction or serving as 
the curriculum specialist in reading, math or early childhood education. In some cases, there was 
additional personnel or more staff time provided to schools through the support of federal 
programs such as Title I Reading First. The activities of school-based support staff members are 
described further in Exhibit 3-3. 



Exhibit 3-3 

Activities of School-Based Support Staff Members 

One key to the use of data to improve instructional practice at the elementary level in this 
large district (District 4) is the availability of full-time instructional advisors (supported by Title I 
funds) and certified data system trainers. Instructional advisors combine expertise in 
accessing data, understanding data, and designing instructional practices to address student 
needs identified by available data. Certified trainers are school-based staff members who are 
trained by the district to support the use of the district’s data system. Trainers are involved in 
ongoing professional development by the district and an annual full-day certification update 
during the summer. Certified trainers initially provided training to their colleagues on how to 
get data out of the system, then moved to how to read data, and are now focusing on talking 
about data, using data and reviewing data. The extent to which the certified trainer role is 
combined with the instructional advisor role (not possible in every school) affects the 
availability of support to school staff members. That is, when certified trainers are full-time 
teachers, the time available for supporting their colleagues is more limited. 

Two large districts have instituted a combination of school-based support staff and district 
policies such as common planning periods focused on data discussions to encourage data- 
informed decision-making practices at their schools. 

District 2 is focusing its efforts on school-based professional development through teams of 
specialists provided to each school. There is a full-time technology specialist in every school 
who has responsibility for training school staff members on using the district data system as 
well as on the use of technology in instruction and the use of assessment tools; the specialists 
have received training on the district benchmark assessments. (Principals and secretaries are 
also trained on how to access the system.) In addition, there are assessment and instructional 
coaches who support school staff members. All high schools have full-time assessment 
coaches who work with teacher leaders and principals on data access, data interpretation and 
instructional planning. The assessment coaches also help manage testing systems. There are 
20 instructional coaches who provide full-time support to the neediest schools, while other 
schools share a coach. Instructional coaches have expertise in curriculum and provide 
teachers with instructional strategies based on test results. Coaches also train and support 
teachers on how to use the district data system and analyze their data. All district elementary 
schools close early on Monday afternoons so teachers can meet either within their grade level 
or across grade levels to examine data and collaboratively plan instruction. 

District 5 provides literacy and math coaches to all its elementary schools (although their 
numbers are shrinking because of declining funding from federal programs). Coaches are 
responsible for clarifying data reports with teachers, reviewing data trends, and modeling 
lessons and practices for teachers. Principals are required to print out and deliver data reports 
to their teachers on their students’ academic performance. The district has incorporated data 
discussions into weekly teacher common planning times (one hour every week). In 
elementary schools, these common planning times almost always involve reading data 
displays, interpreting data, and using data to change classroom practices. Coaches also hold 
after-school workshops to look at school data. 
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Professional Development and Technical Support 

Teacher professional development focusing on data analysis skills or data-informed 
decision making is a relatively recent phenomenon, but districts appear to be responding actively 
to the need for this kind of teacher support. Almost three-quarters (73 percent) of all districts 
responding to the 2006-07 NETTS district survey indicated that they supported professional 
development in the past 12 months “to help teachers and administrators in data-driven decision 
making.” 

Recent years have seen an increase in the proportion of teachers receiving professional 
development on data-informed decision making. On the 2005 NETTS teacher survey, 30 percent 
of teachers said that they had received professional development from their school on this topic, 
and 9 percent said they had received this kind of support from another source. On the 2007 
survey, the proportions receiving professional development on data use rose to 43 percent of all 
teachers for professional development from one’s school and 15 percent for development from 
other sources. 

Teacher survey responses suggest that teachers receiving professional development in this 
area view it as beneficial. On the 2006-07 NETTS teacher survey, 39 percent of teachers with 
access to a data system reported that the professional development they received about data- 
informed decision making had prepared them to use data to improve student achievement. 

At the same time, most teachers reported that they would benefit from additional 
professional development on data-informed decision making (see Exhibit 3-4). Eifty-eight 
percent of teachers with access to a data system thought that additional professional development 
on how to develop diagnostic assessments for their classes would be beneficial, and 55 percent 
said that they would like additional professional development on adjusting the content and 
approach used in their class in light of student data. Almost half (48 percent) reported the need 
for more professional development on the proper interpretation of test score data, and over a 
third (38 percent) for more professional development on how to formulate questions that can be 
addressed with data. Half (50 percent) reported that they could benefit from additional 
professional development on how to identify types of data to collect to monitor school progress 
against goals for improvement, and 44 percent reported a need for more professional 
development on the mechanics of using their data system. Over a third (37 percent) of teachers 
with access to a data system said that they would like professional development on techniques 
for collaborating with colleagues on the use of data. 



As noted earlier, only 33 percent of teachers reported feeling capable of forming queries of the data system 
available to them. It may be that some teachers feel that they can formulate questions that can be addressed with 
data (and hence do not see a need for more professional development on this topic), but find their data system 
difficult to use because of interface issues. 
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Exhibit 3-4 

Topics for Additional Teacher Professional Development 



Topic 


Percentage of 
Teachers Who 
Wanted More 
Professional 
Development 


How to develop diagnostic assessments for your class 


58 


How to adjust the content and approach used in your class in light of 
student data 


55 


How to identify types of data to collect to monitor school progress 
against goals for improvement 


50 


Proper interpretation of test score data 


48 


The mechanics of using the electronic data system 


44 


How to formulate questions that can be addressed with data 


38 


Techniques for collaborating with colleagues on the use of data 


37 



Exhibit reads: Among teachers with access to an electronic student data system, 58 percent indicated 
that they would benefit from additional professional development on how to develop diagnostic 
assessments for their class. 

Source: 2006-07 NETTS teacher survey. 



NETTS survey data also show that teachers’ desire for professional development depends 
on their personal confidence in using a data management system, the perceived support for 
system use, and whether or not their school is making adequate yearly progress (AYP). For 
example, a significantly higher percentage of teachers in schools not meeting AYP indicated that 
they would benefit from additional professional development in six of the seven types of training 
listed in the survey compared with teachers in schools making AYP. Teachers who express a 
lower confidence in their ability to interpret data and to use a data management system are more 
likely to believe that they could benefit from further professional development. 

All case study districts provided some form of districtwide professional development on 
data use, but it varied in terms of who received the training as well as the training content, 
duration and format. For example. Exhibit 3-1 (presented earlier in this chapter) describes how a 
district provides school leadership teams with the opportunity to come together annually to 
review and discuss their state test data and provides them with the tools to carry out this process 
on an ongoing basis to monitor their school improvement goals. In another district, university 
staff members were brought in to provide professional development to kindergarten through 
grade 3 teachers on how to disaggregate and interpret early literacy assessment data. An example 
of district-supported professional development that combines an online professional 
development system and in-person support is provided in Exhibit 3-5. 



The survey included a block of questions asking teachers to reflect on their use of data management systems using 
a 5-point Likert response scale. Additional information on these data can be found in Teachers ’ Use of Student 
Data Systems to Improve Instruction 2005 to 2007 (U.S. Department of Education 2008b). 
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Exhibit 3-5 

Professional Development Online and In Person 

Professional Development Online (PDO) was built jointly by District 3 and a private company. 
This electronic system delivers training on demand to all district staff members and allows 
them not only to view their professional development plan for the year but also to manage 
their training. The district includes required courses in the system, and staff members can 
monitor their course-taking progress. Professional development information is tailored to each 
individual staff member according to his or her position through targeted announcements, 
information and highlighted displays of required and recommended courses. In addition, the 
PDO Web site includes training on how to use the system (PDO Help), professional 
development resources, and a list of professional development conferences. 

As part of its ongoing professional development activities, the district also assigns a staff 
development teacher to every school. He or she focuses on helping teachers improve their 
instructional practice. Staff development teachers in an elementary school, for example, might 
meet with the third-grade teachers as they are planning a lesson, observe a third-grade 
teacher, work on instruction and lesson planning, or model a lesson for some teachers. Staff 
development teachers also work with teachers to examine and analyze data and to design 
instructional programs based on data. 



Three of the case study districts were focusing their professional development activities on 
data use primarily at the school level. The presence of school-based support staff members 
provides opportunities for professional development to be ongoing and tailored to individual 
teacher needs. As noted above, six case study districts provided some sort of school-based staff 
to support data use. Their role included providing professional development to teachers through 
mentoring and serving as a resource to teachers on how to analyze and interpret data. Schools 
also spent some of their professional development time on analyzing school- and grade-level 
data, supported by informal data-related discussions at grade-level or department meetings. 

One district was exploring the issue of school- versus district-based professional 
development, while two districts planned to maintain a balance of district- and school-based 
training. Another district was moving toward more tailored training on how to use the district 
data system. When user statistics indicated that there had been a decline in use of the system, the 
district decided to offer professional development to individual schools based on teacher needs 
(e.g., how to extract the data they need from the system). 

Tools for Acting on Data 

Data systems can help promote data-informed decision making by providing tools to help 
teachers improve decisions about instructional practice. Some of the resources provided include 
instructional materials, model lesson plans, and formative assessment results linked to 
frameworks and curriculum guides. Data from the case studies suggest that these types of tools 
are currently not common resources in district systems. Only three case study districts had data 
systems that provided such resources, and one district was developing a system with this 



The 2007-08 NTA district survey will provide prevalence estimates for tools for acting on data (Questions 7 and 
8) and the frequency with which district staff use their data systems to carry out data-driven decision making 
activities (Questions 18 and 19). 
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capacity. In one of the three districts, its commercial data system offered modules that included 
curriculum and lesson planning features along with an assessment bank that allowed the district 
to administer benchmark tests tied to state standards and to display results to show which 
students passed a particular objective, generating skills and item analysis reports. In another 
district, the data system contained curriculum information aligned with state and local standards, 
instructional frameworks for teachers, and a curriculum guide to help teachers with their pacing 
and content coverage. Teachers could access assignments and lesson plans online so they could 
share resources to supplement instructional materials. 

To link formative test data with instructional resources, the data system must have both 
components. Only four of the nine case study districts had benchmark assessment data on their 
systems (one additional district had some of its benchmark data on the system), and only one of 
these districts had instructional resources on the system that were aligned with the assessments. 
School staff members in several districts used systems outside the district for this purpose. For 
example, one of the districts used a Northwest Evaluation Association (NWEA) test as a 
benchmark assessment, and the NWEA Web site provides instructional resources tied to each 
.skill area in its assessments. 

One of the barriers illuminated by the case study data was the need for greater alignment 
between the district curriculum and benchmark assessments. Eack of alignment presented 
challenges to districts trying to help schools link formative assessment data to instructional 
practice. One case study district had undergone a curriculum audit by the state that identified as a 
weakness the lack of a clear, comprehensive curriculum aligned with standards. As a result, the 
district was developing a uniform curriculum linked to state standards and supported by 
benchmark assessments that were tied to both the curriculum and state standards. The audit was 
also the impetus for the district’s selecting a data system with a curriculum management 
component so instructional resources would be available to teachers for skill areas in which their 
students are identified as being below proficiency on the basis of benchmark data. Until the new 
district curriculum is completed, teachers cannot take full advantage of this component. In 
another district, there was a common districtwide curriculum, but the benchmark assessments 
were not well aligned with the curriculum; the math assessment did not follow the district 
curriculum scope and sequence, and the language arts assessment did not cover all the skills in 
the district curriculum, making it more challenging to support instructional decision making. The 
district reported that these alignment problems would be rectified by a new data system that was 
under development. 

Summary of District and Schooi Supports 

Case study districts’ supports for data- informed decision making are summarized, by 
district, in Exhibit 3-6. Eor each type of support, a district was given a rating: the support was 
widely available in the district (coded as 2); the support was partially present, for example, 
professional development provided to some staff members but not all (coded as 1); or the support 
was not currently present in the district (coded as 0). The districts are ordered by size, from 
largest to smallest. To some extent, the larger districts (those with student enrollments of 25,800 
or more) provide a greater number of supports than the medium-size districts. The mean number 
of fully present supports in very large and large case study districts is 5, compared with an 
average of 3 in medium-size districts. But there are exceptions to the general pattern. District 1, a 
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very large distriet of more than 130,000 students, provides the smallest number of widely 
available supports because of recent resource constraints (e.g., budget cuts eliminated 
instructional coach positions). District 8, a medium-size district, has benefited from external 
funding sources and expertise (e.g., EETT funds, regional consortia) and provides the largest 
number of supports compared with the aforementioned very large district. 

Exhibit 3-7 summarizes the supports available at each case study school, organized by 
district. Each school was rated on the extent to which each type of support was available at the 
school: the support was fully present (coded as 2); the support was partially present, for example, 
a principal who occasionally worked with teachers (coded as 1); or the support was not present 
(coded as 0). Analysis indicated that there was greater consistency within districts than within 
school type (high, emerging, typical) across districts. The average number of fully present 
supports was 3.89, 4.11, and 3.78 for high, emerging, and typical schools, respectively. Averages 
across schools within individual districts ranged from 1.33 to 5.33. This pattern suggests that 
districts play a pivotal role in determining the level of support that teachers receive for the use of 
data systems and making data-informed decisions. 
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Exhibit 3-6 

Case Study District Supports for School Use of Student Data 



Support 


District 

No. 


1 


2 


3 


4 


5 


6 


7 


8 


9 


District 

Size 


VL 


VL 


VL 


L 


L 


M 


M 


M 


M 


Strong district leadership 
for data use 


1 


2 


2 


2 


2 


2 


1 


2 


2 


Availability of data 
disaggregated by student 
groups and skill levels 


1 


2 


2 


1 


2 


1 


2 


1 


1 


Districtwide data-informed 
decision-making activities 


0 


0 


1 


2 


1 


0 


0 


2 


0 


Professional development 
for data system 
implementation 


1 


2 


2 


2 


1 


2 


1 


2 


2 


Provides school-based 
staff members to support 
data-informed decision 
making 


0+ 


2 


2 


2 


2 


2 


0 


2 


0 


Gives teachers direct 
access to extract data 
from system 


1 


2 


2 


2 


0 


0 


0 


1 


2 


District benchmark or 
formative assessment 
data on the system 


2 


2 


2 


0 


2 


1 


2 


1 


0 


System includes links 
from assessment results 
to instructional resources 


0 


0 


2 


0 


2 


0 


0 


0 


0 


Students can take 
assessments online 


0 


2 


1 


0 


0 


0 


0 


0 


0 


Total number of supports 
fully present 


1 


7 


7 


5 


5 


3 


2 


4 


3 



Exhibit reads; In District 1, strong district leadership for data use was partially present in 2006-07 — that is, only 
some departments (not all) were strong advocates for data use. 

Notes; District size categories include very large (VL), with more than 130,000 students, large (L), and medium 
(M). Codes indicate the degree to which each support was present in the district; 0 = not present, 1 = partially 
present, 2 = fully present/widely available. The + indicates that the district used to provide school coaches, but this 
position was not present in 2006-07 because of budget cuts. 
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Exhibit 3-7 

Case Study School Supports for Use of Student Data 



School Support 


District 1 


District 2 


District C 




District 4 


District 5 


H 


E 


T 


H 


E 


T 


H 


E 


T 


H 


E 


T 


H 


E 


T 


Principal sets example for data 
use 


2 


2 


2 


2 


2 


1 


1 


2 


0 


2 


2 


2 


2 


2 


2 


Principal works with teachers 
on data 


2 


2 


2 


2 


2 


1 


0 


2 


0 


0 


2 


2 


0 


0 


0 


School-based staff members 
available to support data use 
and/or other school leaders 
work with teachers on data 


0 


1 


0 


2 


2 


2 


2 


2 


2 


2 


2 


2 


2 


2 


2 


Specific procedures in place to 
discuss/act on data 


1 


0 


0 


1 


0 


0 


2 


1 


1 


1 


1 


1 


1 


2 


2 


School has own professional 
development on data use 


2 


1 


2 


2 


2 


2 


2 


2 


2 


1 


2 


2 


1 


1 


1 


Time allotted during school 
day for teachers to work with 
data 


2 


2 


2 


2 


1 


2 


2 


2 


2 


2 


2 


2 


2 


2 


2 


Total number of supports fully 
present 


4 


3 


4 


5 


4 


3 


4 


5 


3 


3 


5 


5 


3 


4 


4 



Schooi Support 


District 6 


District 7 


District 8 


District 9 


H 


E 


T 


H 


E 


T 


H 


E 


T 


H 


E 


T 


Principal sets example for data 
use 


2 


2 


1 


0 


2 


0 


2 


2 


2 


2 


2 


2 


Principal works with teachers 
on data 


2 


2 


0 


0 


1 


1 


2 


2 


2 


2 


2 


0 


School-based staff members 
available to support data use 
and/or other school leaders 
work with teachers on data 


2 


2 


2 


2 


0 


0 


2 


2 


2 


1 


0 


0 


Specific procedures in place to 
discuss/act on data 


1 


1 


1 


1 


0 


0 


2 


2 


2 


1 


1 


1 


School has own professional 
development on data use 


2 


2 


2 


1 


0 


1 


1 


1 


2 


2 


0 


2 


Time allotted during school 
day for teachers to work with 
data 


2 


2 


2 


2 


1 


2 


2 


2 


2 


2 


2 


2 


Total number of supports fully 
present 


5 


5 


3 


2 


1 


1 


5 


5 


6 


4 


3 


3 



Exhibit reads: In District 1, at the high-data-use school, the principal set an example for data use and also worked with teachers on using data. 

Notes: Codes indicate degree to which each support was present in the school during 2006-07 : 0 = not present, 1 = partially present, 2 = fully present. Schools 
were defined by district staff members for their level of data use: H = high data use, E = emerging data use, T = typical data use for the district. 




4. Teachers’ Preparation for Data Use 



As noted in the preceding chapter, staff members in case study districts expressed 
reservations about teachers’ ability to obtain student data from systems or to make sense of 
student data reports provided to them. Understanding the nature of teachers’ proficiencies and 
difficulties in this arena is important for the design of both data-informed decision-making 
processes (e.g., whether or not to have specialists serve as mediators between teachers and 
data systems) and of teacher education and professional development programs. To provide 
insights into areas of teacher strength and weakness with respect to using data, project staff 
incorporated data scenarios into case study activities as a context for probing teachers’ ability 
to engage in various aspects of data interpretation and use. The data scenarios were presented 

27 

individually to six teachers at each school. 

Development of the Data Scenarios 

The rationale behind the development of the data scenarios was to have a standard set of 
prompts that could be used to elicit teachers’ thinking about student data and reveal the 
concepts and skills that they can bring to data-informed decision making. Study staff 
members assembled a group of internal and external experts in assessment and data-informed 
decision making, including two assessment experts, an expert on the use and functionalities of 
student data systems, a leading researcher in the area of mathematics education, and two 
researchers who had performed doctoral or postdoctoral research on the use of student data 
systems to inform educational decision making. 

Working with this group, the study’s principal investigator first identified major 
processes involved in using student data to inform school-level decisions. These processes 
included the following: 

• Question posing. In cases where a data system is available for exploration and 
educators must decide how to use it in a way that can improve their practice, the 
educator needs to be able to frame a question that can be addressed with available 
data. Doing so requires an appreciation of the kind of question that can be 
answered with data and alignment between the question’s intent and the measure 
and student group specified in the data query. 

• Data comprehension. Data are typically displayed in tables, graphs or printouts, 
which can be quite complex, and finding the desired data element is not a trivial 
matter. Further, the educator needs to understand the data display so he or she can 
answer the question, “What do the data say?” 

• Data interpretation. Beyond the issue of literal data location and comprehension, 
particular elements of data comprehension require at least a qualitative 
understanding of basic concepts in statistics; for example, an educator who does 



Before the site visit, principals were asked to select six teachers to participate in the data scenario interviews: 
three teachers who actively use data to inform decision making and three teachers who represent average data 
use within the school. 

This group comprised TWG members Jeff Wayman and Ellen Mandinach, as well as Jere Confrey (then of 
Washington University) and SRI staff members Eva Chen, Geneva Haertel, and Viki Young. 
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not appreciate the effect of an extreme score on a sample mean can be misled by 
data sets with outliers. 

• Data use. For student data and data systems to have a positive influence on 

instruction, data must be not only located and properly interpreted but also linked 
to appropriate instructional options. The educator must have alternative 
instructional strategies or choices available and a willingness to use data to inform 
decisions. 

For each of these components of data-informed decision making, expert-group members 
identified specific skills and concepts that teachers should have to execute this aspect of data 
use successfully. 

After identifying key skills and concepts for using data, the group brainstormed 
examples of situations or questions that would call on each of the concepts and skills in 
question. Screen shots from actual data systems and questions that had been used in prior 
research or teacher education also were reviewed by the group as possible item models. 

The list of priority concepts and skills developed by the expert group and the example 
situations calling on those concepts and skills were then used by the research team as a 
starting point for generating eight data scenarios (see Appendix B). Draft scenarios were 
reviewed by an assessment expert and a mathematics education expert from the group for 
plausibility, accuracy and alignment with the identified skills and concepts. 

One of these reviewers pointed out that data comprehension items could be classified into 
two types. Some items called simply for finding the correct data element in a display. These 
were subsequently labeled data location items. Others added a degree of difficulty by 
requiring the manipulation of displayed data (such as the addition of two percentages or the 
computation of a proportion) to answer a question about what the graph or table showed. 
After the review, the term data comprehension was reserved for this latter item type. Exhibit 
4-1 shows the final set of concepts and skills identified by the group of experts. 

The assessment scenarios were pilot-tested first with former teachers on SRFs staff and 
then with practicing teachers, with revisions made after each round of pilot testing. To cover 
all of the identified skill areas without extending interviews to an intolerable length, 
developers created three protocol “forms,” each with five of the eight scenarios. The amount 
of content on the various forms was balanced to achieve roughly equivalent average 
administration times, estimated at 30 minutes each. 
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Exhibit 4-1 

Data-Informed Decision-Making Components, Concepts and Skills 



Component 


Target Concept/Skill 


Question Posing 


Aiigns question with purpose and data 


Forms queries that iead to actionable data 


Understands iterative nature of data exploration 


Understands value of multiple measures 


Data Location 


Finds relevant cells in a complex table 


Finds relevant cells in a complex figure 


Data Comprehension 


Manipulates data from a complex table or graph to support 
reasoning 


Maps between data in a table and a prose representation of 
the data 


Maps between data in a figure and a prose representation of 
the data 


Understands a histogram as distinct from a bar graph 


Interprets a contingency table 


Distinguishes between longitudinal and cross-sectional data 


Evidences data comprehension monitoring (metacognition) 


Data Interpretation 


Understands the advantages and disadvantages of using 
disaggregated subgroup data vs. individual student data 


Attends to distribution and extreme quartiles, not just mean or 
proportion above cut score 


Appreciates effect of a few extreme scores on the mean 


Realizes that more items on a scale or members in a sample 
produce more precise, reliable estimates 


Understands measurement error and variability; results not 
identical on every testing 


Understands that student cohorts differ from year to year 


Data Use 


Understands how to differentiate instruction based on data 


Seeks subscale and item data that can be mapped to 
curriculum 


Understands value of formative assessments 
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Scenario Administration 



Within each of the 27 schools participating in the 2006-07 case study site visits, 
principals were asked to identify six teachers for participation in interviews and focus groups. 
They were asked to choose three teachers who were active in school use of data to inform 
instruction and three teachers who were typical of the school teaching staff in this regard. The 
data scenarios constituted the second part of the individual interviews for these teachers. For a 
majority of schools, six teachers participated in the study as planned. For a few schools, it was 
possible to meet with only two to four teachers because of schedule conflicts. In total, 147 
teachers from 27 schools responded to a set of data scenarios in spring 2007. Each form 
included scenarios addressing multiple components of data-informed decision making. 
Scenarios contained a combination of open-ended and structured-response items. The data 
scenarios varied in the number of items they contained, but each included items that could be 
scored using either dichotomous or partial-credit rubrics. Examples of items representing 
each stage in using educational data for decision making are presented in Exhibit 4-2. All 
items and rubrics, as well as item allocations to forms, are included in Appendix B. 



Exhibit 4-2 

Examples of Items Related to Each Component 



Component 


Item Prompt 


Full-Credit Response 


Question 

Posing 


So now in January 2007, what specific 
data would you want to get from this 
system to help you decide how to 
improve your fourth graders’ 
performance? 


Teacher picks a logical group and 
selects a logical measure for that 
group. 


Data Location 


What was Oak School’s average Total 
Math Score in 2003-04? 


Teacher provides correct answer 
from graph within 5-point range. 


Data 

Comprehension 


Oak School’s progress in narrowing the 
grade 4 math achievement gap with the 
rest of the district has been in problem 
solving rather than computation. 
(Teacher must agree or disagree and 
explain reasoning.) 


Agrees. Teacher describes data 
illustrating that the gap between 
Oak School and district scores 
decreased more for problem 
solving than for computation. 


Data 

Interpretation 


These data suggest that next year’s 
third-grade Asian/Pacific Island girls will 
score better than other third graders on 
this test. 


Teacher disagrees and explains 
that students in a subgroup vary 
from year to year and you cannot 
generalize based on just a single 
Asian student the prior year. 


Data Use 


Teachers should obtain a detailed 
breakdown of last year’s test results by 
item or content standard. 


Teacher agrees with a 
reasonable explanation about 
why having detail is important. 
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Analysis of other aspects of the protocols is ongoing, using a qualitative coding scheme to capture additional 
information about teachers’ thinking. Findings from the qualitative coding will be presented in the final report. 
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Measures 

Multiple questions were associated with each data scenario. Most questions were 
designed to get at a specific concept or skill and to have right and wrong answers. A teacher’s 
response to each of these questions was assigned a 1 if correct and a 0 if incorrect. For some 
items, there were intermediate or partially correct responses for which a .50 was assigned. In 
addition, some questions were more open-ended, deliberately designed to elicit teachers’ 
thinking. Responses to these questions are not included in the scores reported here. Teacher 
responses to these open-ended questions have been transcribed and are being coded for 
evidence of types of data-informed decision-making skills (e.g., indications of comprehension 
monitoring, descriptions of how data would be used to differentiate instruction). 

Procedures 

Interviewer training. Site visitors were involved in a full-day training session that 
included an overview of the study’s conceptual framework, the data systems used by each 
district, and the various data collection activities, including the data scenarios. Site visitors 
were shown a video of the administration of the data scenarios to illustrate proper 
administration techniques and then given the opportunity to practice administering the data 
scenarios to one another. This activity was followed by a discussion period to answer 
questions that arose as a result of the practice session. 

Data scenario interviews. Teachers participated in 45-minute interviews with two 
researchers. Approximately the first 15 minutes of the interview involved questions 
concerning the teacher’s personal experience with the district’s data system, including 
decisions he or she had made on the basis of student data. Teachers then responded to items 
from one of the three data scenario forms. They were told that the study wanted to investigate 
how different kinds of data displays are understood by teachers and were asked to think out 
loud as they looked at the various data presentations and responded to questions about them. 
When the data scenarios were presented, one researcher was responsible for asking teachers 
questions from the assigned data scenario form while the other researcher took notes. 
(Teachers were randomly assigned to forms before the interview.) Each form was 
administered to two teachers within a school, if all six teachers from the school participated in 
the study. Teachers were provided with copies of the graphs, tables and screen shoots 
included in each scenario. Teachers were also provided with paper, pencils and calculators 
they could use as they wished (e.g., to carry out basic arithmetic calculations). All interviews 
were audio-recorded and transcribed to facilitate later analysis. 

Scoring. Before scoring, researchers reviewed each transcript to identify the beginning 
and end of the discussion of each item. Each item segment was coded with an item 
identification number in ATEAS.ti, a qualitative data analysis program. ATEAS.ti was used 
to produce data reports by item (i.e., all responses for a given item) to facilitate the scoring of 
a single item at a time. Eor each item, raters reviewed the rubric using a sample of transcripts 
and then refined the rubric on the basis of the range of teacher responses to the item. 

Two raters scored each item, and at least 30 percent of all item responses were double- 
coded to permit the examination of interrater agreement. Interrater agreement was 90 percent 
or higher for 29 items, between 80 percent and 90 percent for four items, and 75 percent for 
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one item. All discrepancies between raters’ scores were resolved by consensus between the 
raters. 

For two of the scenarios (those focusing on Question Posing and Data Use), all but one 
or two of the scenario-based questions were open-ended. The responses to these and other 
open-ended questions have been transcribed and are being coded for evidence of the skills and 
concepts of interest. 

Preliminary Results and Findings 

Exhibit 4-3 shows the mean score (percentage of full credit) for each data component for 
which five or more items could be scored as right or wrong. The data scenarios themselves 
and the distribution of responses to those questions that were scored as correct or incorrect are 
included in Appendix B. The proportion of respondents giving correct answers for each 
assessment item and information concerning the reliability of the data scenario assessment 
items as a set are included in Appendix C. 



Exhibit 4-3 

Frequency Distribution and Mean Percentage Correct for Scored Items Overall 

and Within Each Data Component 



Component 


Number of 
Scored 
Items 


Mean Number of 
Respondents per 
Item 


Mean 

Percentage 

Correct 


Standard 

Error 


Total score 


33 


84 


67 


1.13 


Question Posing 


1 


85 


NA 


NA 


Data Location 


10 


72 


81 


1.61 


Data Comprehension 


16 


90 


64 


1.53 


Data Interpretation 


5 


88 


48 


2.60 


Data Use 


1 


77 


NA 


NA 



Exhibit reads: The average percentage of a school’s teachers responding correctly across all 33 data scenario 
items scored was 67 percent. 

Note: NA = Insufficient data (fewer than 5 items). 



The data in Exhibit 4-3 suggest that teachers are generally capable of finding specific 
pieces of information in a data table or graph (the mean percentage correct for Data Eocation 
items was 8 1 percent) but that their ability to translate between a statement and a set of data 
declines when they must do computation or make multiple comparisons using the numbers 
provided (for Data Comprehension items, the mean was 64 percent). When teachers are asked 
to go one step further and to reason about data issues that involve basic statistical concepts 
such as variability, measurement error or distribution, their likelihood of making a correct 
inference is still lower (48 percent correct). 

The data scenarios provide insight into the nature of teachers’ misconceptions related to 
use of student data. On Data Interpretation items calling for teachers to derive predictions 
from data, for example, a majority of teachers relied on subgroup means without 
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consideration of the scores for specific students at risk. When looking at a school’s or 
district’s multiyear data, a majority of teachers lacked the understanding that the students in a 
particular grade cohort differ from year to year. When reasoning about the achievement trend 
of an individual school over time, teachers, on average, did not consider districtwide trends 
when interpreting year-to-year changes in the school’s scores. The ongoing qualitative 
analysis of teacher responses will elucidate further teachers’ reasoning about data. 

Comparisons by District 

Because different teachers responded to different data scenarios, depending on the form 
administered to them, teacher scores were aggregated to the sehool level. Mean school scores 
were examined by distriet to determine whether there were any district-level differences in 
performance (Exhibit 4-4). In general, the nine districts appeared comparable with respeet to 
their teachers’ average performance on the data seenarios, but two districts did stand out as 
potentially different. District 1 had the highest mean score, and District 5 had the lowest 
seore. A one-way ANOVA revealed that varianee by district was not statistically significant 
[F(8,18) = 2.41,;? = .06, = .52]. 



Exhibit 4-4 

Total Average Scores for Schools, by District 

0.90 
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0.79 




123456789 

District iD 



Exhibit reads: The total average score for schools in District 1 was 79 percent 
correct. 



Remaining Analyses 

The analyses presented in this report provide initial insight into variation in data- 
informed decision-making skills for schools and districts. The final report will contain 
findings from the qualitative coding of teacher responses to open-ended data seenario 
questions and provide greater detail about the types of knowledge and skills that teachers use 
as they develop questions, read data tables and graphs, interpret data, and use data to make 
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instructional decisions. Also, for the final report, the teacher data set will be expanded with 
additional teachers interviewed in 2007-08 and with responses of small groups of teachers 
responding collectively rather than as individuals to the same or similar scenarios. 

Summary 

The data in Exhibit 4-3 suggest that teachers generally can locate specific information in 
a data table or graph but that they experience increasing difficulty if they need to manipulate 
the data in some way or make comparisons between multiple data points. Necessary 
manipulations required only basic arithmetic (subtraction, division), and teachers were 
provided with paper and pencil and with calculators they could use if they wished. Even with 
these tools, cognitive research would suggest that the cognitive load required for keeping in 
mind the issue at hand, multiple data points, and intermediate products of data manipulations 
have a role in the observed deterioration of teachers’ accuracy (Ayres 2001). Upcoming 
administrations of data scenarios to small groups of teachers will test whether greater 
accuracy in data comprehension is achieved by spreading the comprehension burden across 
multiple individuals. 

In terms of the ability to interpret data, teachers demonstrated limited understanding of 
basic statistical concepts such as variability and reliability. Teachers appeared to have 
particular difficulty (a) using individual students’ scores, rather than their subgroup mean, as 
the basis for predictions and instructional prescriptions and (b) keeping in mind the fact that 
student groups differ from year to year when looking at multiyear data. 
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5. Summary of the Early Findings 



The concept of using data as a basis for their decisions is novel to many school staff 
members, and there are multiple obstacles to institutionalization of this practice. By collecting 
data at the school and district levels, the research team sought a “bottom-up” perspective on 
the quality of the supports for data-informed decision making coming from the state and 
national levels. Policy-making in this area began with the assumption that federal 
requirements and national campaigns (e.g., the Data Quality Campaign, grants to states for 
enhancement of their data systems) would lead to higher-quality state data systems, which in 
turn would have a positive influence on the use of data in districts, schools and classrooms. 

Although federal and state policies have spurred districts and schools to initiate more 
data-informed activity, the study found no evidence of a tightly coupled system. The vast 
majority of the activity around data systems and data use in districts and schools involves 
district, not state data systems, according to the NETTS survey data. Two of the case study 
districts were in states with systems that use a unique student identifier that the district system 
also used, facilitating transfer of data across systems. But for the most part, the hoped-for 
efficiencies to be gained from integrating data systems at the state, district and school levels 
are not apparent from the vantage point of schools and districts. More progress in this area can 
be expected as new state systems come on line. 

Direct district use of a state-provided software system appears to be relatively 
uncommon. This practice was found in only one of the nine case study districts. In this case, 
the state made available a performance improvement mapping system that the district could 
use to obtain test items linked to areas where the school needed to improve. In another case, 
the district was using a state software system for its data warehouse, but the state had since 
moved to a new software system with additional capabilities, and the district was left with an 
“orphan” system. 

Among the nine case study districts (drawn from nine different states), only one cited 
training on data-informed decision making made available through its state education agency. 
In this case, principal training and some funds for laptop purchases were made available by 
the state. 

It should be noted that a tightly coupled system is not necessarily the ideal solution in all 
cases. The fact that no such system was found in any of the case study districts, selected on 
the basis of perceived excellence in data use, suggests that implementation of such systems is 
difficult. Much of the value provided by a tightly integrated system can arguably be provided 
also by less tightly coupled systems. 

Case study informants were more likely to describe federal funds supporting data use 
activities than state funds doing so. A number of federal programs have provided funds that 
have paid for extra staff positions or other activities to promote a data-using culture. In one 
district, for example. Title I funds paid for full-time school-based instructional advisors who 
helped teachers acquire and interpret data; the funds also supported the high-speed Internet 
access needed to use the data system. Another district used some of its Title I funds for 



45 




^0 

professional development on the use of DIBELS data for instructional decision making. 
Schools in three districts used Reading First grant funding for activities such as collecting and 
tracking longitudinal data on primary students’ reading skills and for funding reading coaches 
who helped teachers incorporate assessment data into their practice. An EETT grant to a state 
consortium for school use of data was led by one of the case study districts; the consortium 
has provided professional development on data-informed decision making at no cost to the 
district. 

University partners are an additional class of facilitators. One district is working with a 
literacy project run by a teachers’ college to learn more about kinds of student data that can be 
collected and how they can be used to guide subsequent instruction. Another district had a 
university come in to work with its teachers on how to disaggregate and interpret data from a 
phonological awareness assessment. 

Progress in the Use of Data Systems 

This study encompassed both district and school use of student data systems and their 
data use practices to improve instruction. Across districts and schools in the case study 
sample, use of locally generated data to inform instruction and use of electronic data systems 
containing student scores on standardized tests appeared to be two parallel initiatives rather 
than a single, integrated effort. District and school use of standardized test scores obtained 
from data systems typically focused on accountability concerns and making sure that local 
curriculum and instruction were well aligned with the state’s assessments. Eocally generated 
data often came from instruments connected with an early literacy program or benchmark 
assessments. The two data activities were not necessarily integrated, and schools making 
progress in one of these areas were not necessarily doing much with the other. 

Staff members at four of the nine case study districts reported having identified areas in 
need of improvement, corrections needed in curriculum alignment, or areas ripe for new 
programs on the basis of analysis of data from their data systems. Two of the districts 
described more mature data use, including using data on an ongoing basis to evaluate which 
of their initiatives were working and which were not. One of these districts won a state award 
as one of the most improved districts in the state. 

The NETTS teacher survey and first round of case study data collection make it clear 
that the implementation of data systems and practices for their use in decision making at the 
school level is still in its infancy. Between the two NETTS teacher surveys, the proportion of 
teachers reporting access to a student data system rose by nearly 50 percent. This large a 
change over a two-year period is quite remarkable, but the optimism it engenders must be 
tempered by concerns over the perceived value of the data to which more teachers have 
access. On the latest NETTS survey, the roughly two-thirds of teachers with access to a 
student data system were divided in their perspective on whether or not school use of data had 
paid off: 42 percent agreed that their school had been improved through the use of data, while 
45 percent neither agreed nor disagreed with the statement. Similarly, the case study 

The Dynamic Indicators of Basic Early Literacy (DIBELS) are a set of standardized, individually administered 
measures of early literacy development. They are designed to be short (1 -minute) fluency measures used to 
regularly monitor the development of pre-reading and early reading skills. 
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interviews suggest that even in districts with a reputation for leadership in using data, 
electronic data systems are barely influencing classroom-level decision making. However, 
more progress can be seen in the use of data, coming from other sources such as diagnostic 
tests included in early reading programs, to shape instruction. 

Staff members at all 27 case study schools described the use of student data that was not 
contained on their data system to guide instruction. Frequently mentioned were data from 
early reading assessments given in primary grades (for which state assessment data are 
usually unavailable), performance reports from computer-based instruction, running records, 
and examinations constructed by school departments. 

Lessons Learned for Implementation 

The reflections of the case study informants and the pattern of activity across their 
districts and schools with different practices suggest a number of guidelines for schools and 
districts embarking on the implementation of data-informed decision making. 

1. Districts are finding that providing school-level data coaches is an important support for 
school-level use of data to inform instruction. In the majority of case study districts, some 
or all schools had on-site staff members designated to help teachers retrieve data from the 
data system, interpret data and make instructional decisions based on data. Not all districts 
started out with this intention, but it appears to be an emerging best practice that both 
encourages more use of data and lessens the likelihood of misinterpretation of data. 

2. A common curriculum and curriculum-aligned benchmark assessments increase the 
likelihood that school staff members will make extensive use of a district’s data system. 
State assessments are typically administered once a year, and school review and reflection 
on the resulting data is typically also a once-a-year event. Teachers in all nine case study 
districts looked more frequently at interim, benchmark or end-of-course assessments. In 
some cases, these data were available on a district-provided system, but in many cases they 
were accessed on a test developer Web site, either as reports generated by the assessment 
software itself or through a school-developed database. To the extent that the district data 
system contains more current data that teachers find relevant to their instructional 
decisions, teachers will have a greater motivation to use the system. 

3. If teacher use of data is the goal, then it is desirable to have curriculum and instruction 
staff members involved in the initiative. The use of student data to inform instruction is 
necessarily a systemic effort, linking assessment and accountability, professional 
development, information technology, and curriculum and instruction functions that are 
typically separate offices within medium-size and large districts. All of these offices need 
to support the initiative. In two of the case study districts, the data use initiative was led by 
the assessment and accountability office, and other district offices were not deeply 
involved. In these cases, the data system was not being used to shape instruction. 

4. Teacher buy-in for the data system and its use should be sought early and maintained 
continuously . Attention to teacher buy-in was not uniform across the case study districts. 
On one end of the continuum were districts that solicited input from a few principals or 
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teachers regarding what they wanted in the system and then proceeded to procure and 
adopt one. At the other end of the continuum was a district that instituted a nine-month 
process of convening stakeholder groups, including teachers, to help select the new system 
and involved the union in the media campaign for its launch. Several districts that had not 
involved teachers early on were dealing with teacher suspicions that the data system would 
be used to identify weak teachers and that data would be used against them. 

5. Professional development should include training on how to interpret data and how to 
translate data into changes in instructional practice. Teachers in seven of the nine case 
study districts cited insufficient training on how to engage in data-informed decision- 
making practices, how to read and interpret the data that are given to them, and how to 
translate data into actionable plans for instruction. Teacher responses in the data scenario 
interviews suggest that these concerns are well-founded. Teachers could read data tables 
and graphs but had difficulty framing a question that could be answered through a data 
system query, and when they did have data in front of them, they had a hard time drawing 
defensible interpretations or inferences based on the data. On the NETTS teacher survey, a 
large proportion of respondents felt they could benefit from additional professional 
development around the use of data: 58 percent thought they could benefit from 

o 1 

professional development on how to develop diagnostic assessments for their students, 55 
percent on how to adjust the content and approach used in their class on the basis of data, 
50 percent on what data to collect to monitor school progress against improvement goals, 
and 48 percent on how to interpret data. 

6. District policies should be examined to identify and remove policies and procedures that 
undermine teachers’ use of data to inform instruction. Despite a commitment to the 
promotion of teachers’ use of data in making instructional decisions and individualizing 
and optimizing students’ educational experiences, districts often have policies in place that 
run counter to such efforts. An example is the imposition of district pacing plans that do 
not provide time for individualization or selective reteaching based on benchmark or 
formative assessment findings. On the NETTS Teacher Survey, of those teachers with 
access to a student data system, 60 percent said that no matter what the system tells them 
about their students’ learning, they must keep up with state or district pacing plans. 

7. School leaders need to build teachers’ mutual trust to a point where teachers are 
comfortable working with colleagues to examine data that reflect on their teaching 
performance. Much of the examination of data in case study schools occurred in small 
groups — during common planning time or grade-level or department meetings. This 
practice was particularly well ingrained in the schools identified as “high use” by their 
districts. Staff members in seven districts said that data-informed decision-making 
practices had moved school staff members toward more open conversations about 
instructional practice and increased opportunities to learn from one another. One principal 
noted, “Using data eliminates distracters and keeps everyone focused on where they need 
to go. Data also helps to eliminate some of the interpersonal issues that can arise; it’s not 
about who you are, it’s about what the data shows.” Some schools are actually 



Teachers are being encouraged to use diagnostic assessments but often express frustration with the fact that 
they do not have diagnostic instruments available to them that match their instructional program. 
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disaggregating assessment data by teaeher to find areas where there is a marked differenee 
across classrooms so teachers can learn from peers with the best outcomes in that area. 
Clearly, such activity requires a high degree of mutual trust, a characteristic that school 
reform research has found to be present in urban schools that make significant 
improvements (Bryk and Schneider 2002). The principals at two of the case study schools 
noted that they needed to spend several years building up an atmosphere of trust and a 
“blame-free” culture before their teachers could look at data together honestly. 

The NETTS teacher survey data also underscore the importance of peer support in 
creating a culture of data use. Teachers who reported feeling well-supported by their 
colleagues in their efforts to use data were more likely than other teachers to use data in ways 
related to instruction, for example, to identify student skill gaps. 

Data-informed decision making is not a simple intervention not only because it involves 
so many aspects of education (e.g., assessment, curriculum, accountability, information 
technology) but also because it requires fundamental improvements in the degree of mutual 
trust and changes in the way teacher time is used. The survey and case study data presented in 
this interim report suggest that the movement to incorporate student data in local education 
decision making is real (many districts, school leaders and teachers are making good-faith 
efforts), but there is a significant distance to go before it becomes well executed in practice. 
The picture in 2006-07 was one that showed various pieces of the complex requirements 
getting put in place in different locations but no single school or district demonstrating a 
totally integrated, consistent and pervasive continuous-improvement process. Mutual trust 
may prove to be the glue needed to hold together the district and school practices that involve 
using data to improve instruction and achievement. 
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Exhibit A-1 
Definitions of Terms 



Term 


Definition 


Data systems 


Electronic information systems to assist in the organization and 
management of data. They consist of hardware and software that 
provide many different functions to users such as storing current and 
historical data, rapidly organizing and analyzing data (e.g., examining 
relationships within data, specifying data subgroups), and developing 
presentation formats or interfaces. 


Interoperability 


The ability of different data systems or software packages to 
communicate with one another. 


Software applications 


Software applications consist of a wide range of specialized products 
to facilitate access to data, data analysis and interpretation, and 
presentations of data (e.g., formatted reports, graphing functions). 


Schools Interoperability 
Framework (SIF) 


Data exchange standards for K-12 education software developed to 
promote data sharing between different applications without further 
software intervention. SIF was launched in 1998 as a nonprofit 
initiative of the software and information technology industry to 
ensure that K-12 instructional and administrative software 
applications work together more effectively. 


Data-informed decision 
making 


A process that integrates the analysis of educational data, typically 
stored in educational data systems, to support decisions intended to 
improve teaching and learning at the school and classroom levels. 
The practice entails regular data collection and ongoing 
implementation of improvements. 


Longitudinal student 
data 


Data on individual students collected over time that allows users to 
compile an academic history for each student. This type of data 
enables more robust analyses of student performance to help 
differentiate instruction and improve student achievement. 


Query tool 


Software that allows for customized and ad-hoc data requests such 
as “drill down” capability to efficiently examine a subset of data at a 
grade, classroom or student level. 


Data query 


A request for specific records from a data system. 


Transaction capture 


Real-time accounting of daily school functions such as attendance 
and school lunch counts. 


Electronic grade books 


Online tools to help teachers manage classroom activities (e.g., 
generating seating charts, recording grades and test scores, 
managing attendance, tracking skills and standards, planning 
lessons, and generating report cards). 


Electronic portfolios 


Online storage of student work samples. 


Electronic 

communication tools 


Tools to facilitate communication of information such as use of e-mail, 
development of Web sites, and electronic discussion groups or 
“message boards” where users from multiple locations can discuss 
issues. 


Formative assessment 


Assessment conducted during instruction to provide feedback that 
can be used to adjust ongoing teaching and learning to improve 
student outcomes. Formative assessment can be contrasted with 
summative assessment, which takes place after a period of 
instruction, to judge how much learning has occurred. 
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A-3 



Exhibit A-2 



Data Available From the State Data System for Each Case Study District in 2007 



District 


1 


2 


3 


4 


5 


6 


7 


8 


9 


Data Elements in State 
Information System 


CA 


VA 


MD 


TX 


Rl 


MA 


NC 


CO 


MO 


Unique statewide student identifier 
that connects student data across key 
databases across years* 


Yes 


Yes 


No 


Yes 


Yes 


Yes 


No 


Yes 


No 


Ability to match individual students’ 
test records from year to year to 
measure academic growth* 


Yes 


Yes 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


State can measure year-to-year 
growth for individual students in any 
subject (state tests in the same 
subject in consecutive years and is 
able to connect data across years) 


No 


Yes 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Student-level enrollment, 
demographic, and program 
participation information* 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Student-level transcript information, 
including information on courses 
completed and grades earned* 


No 


No 


No 


Yes 


No 


No 


Yes 


No 


No 


A teacher identifier system with the 
ability to match teachers to students* 


No 


No 


No 


No 


Yes 


No 


No 


No 


No 


Identifies which schools produce the 
strongest academic growth for their 
students 


Yes 


Yes 


No 


Yes 


Yes 


Yes 


No 


Yes 


No 


Knows what achievement levels in 
middle school indicate that a student 
is on track to succeed in rigorous 
courses in high school 


No 


No 


No 


Yes 


No 


No 


No 


No 


No 


A state data audit system assessing 
data quality, validity and reliability* 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Yes 


Yes 


Yes 


Number of the 10 DQC essential 
elements present* 


7 


7 


3 


9 


6 


8 


7 


6 


4 



Exhibit reads: In District 1, in 2007, the state data system has a unique statewide student identifier that connects student data across key databases across years. 

Note: An asterisk identifies six of the 10 essential elements most relevant to the data issues discussed in the case studies. There are four additional elements not 
listed. 

Source: Data Quality Campaign (2007). 
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Exhibit A-3 

Case Study District Data Systems 

TurnLeaf Achievement Management System serves as the data 
warehouse and assessment system; eSIS is the student information 
system; NWEA online database for analysis and reporting tools and 
instructional materials. 

eScholar is the data warehouse; SIMS is student information system; 
Star_Base for student records; TestWiz managed by dataMetrics 
Software; TPD electronic registrar; PIMS performance improvement 
mapping system from the SEA; EdIine for reporting teacher data to 
students and parents. 

SchoolNet Account (report generation); Assess (assessment bank) 
and Align (data warehouse) modules; Pentamation is the student 
information system; Special Education Manager; discipline and 
attendance systems. 

Education Decision Support Library is the data warehouse (EDSL is 
LEA developed); Benchmark Assessment and Reporting Tool (BART 
was jointly developed by LEA and Princeton Review); Schools 
Administrative Student Information (SASI) from Pearson, Educational 
Curriculum Assessment Resource Tool (ECART is being developed by 
LEA to replace BART). 

SIMS is the state data system and serves as the LEA data warehouse 
(will be replaced by NC WISE); C-CASS for special education data; 
DIBELS online database for analysis and reporting tools (data 
uploaded from handhelds). 

Data Warehouse was developed by the LEA; Instructional 
Management System (IMS designed by ADMIN.COM); Professional 
Development Online (PDO was built by LEA and a private company). 

Registration (Reg was developed by the LEA) is a student information 
system and data warehouse; Socrates CRM (the library data system); 
Phonological Awareness Literacy Screening (PALS) Web site for 
analysis and reporting tools. 

Infinite Campus houses student data and has a suite of reporting tools 
and parent portal; EdGate assessment system is being piloted. 

Cognos (acquired by IMB) is the LEA’s data warehouse; Zangle is a 
Web-based transactional system to enter data (attendance, lunch, 
district test scores); Student Assessment Data System (SADS) 
provides historical test data and standard reports; Encore is the 
special education system (LEA-developed systems). 
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NETTS Survey Data 

The primary survey data included in this report come from the U.S. Department of 
Education’s National Educational Technology Trends Study (NETTS): 1,039 district 
technology directors surveyed during spring 2007 and 2,509 teachers surveyed in spring 2007. 
The teachers were clustered in schools sampled from districts participating in the NETTS 
district survey. The NETTS district sample of 1,039 districts was nationally representative 
with respect to poverty status, student enrollments, and location (urban or rural status). The 60 
largest urban school districts across the country were selected with certainty (i.e., included in 
the sample from the outset). Districts composed entirely of special education schools and 
vocational-technical schools, as well as independent charter schools that are their own 
districts, were excluded from the district sampling frame because of their dissimilarity to 
“typical” districts. The teacher sample was created by drawing a probability sample of 975 
schools from respondents to the district survey, stratified by school type (elementary or 
secondary), and poverty level (high or low). Schools were randomly sampled in proportion to 
the number of teachers and in inverse proportion to district size to produce a sample of 
schools whose selection probabilities were roughly independent of the size of their district’ s 
enrollment. 

Higher-poverty schools were oversampled (233 of the 975 schools) to obtain more 
precise data about their technology use. Eor schools, “higher poverty” was defined as above a 
specified cutoff in terms of the percentage of students who were eligible for the free or 
reduced-price lunch program. The dividing line between higher-poverty and lower-poverty 
schools was selected to ensure that for each school type (elementary, middle or high school), 
there would be the same number of teachers in the higher-poverty and the lower-poverty 
groups, as reported in the National Center for Education Statistics Common Core of Data 
(CCD). Elementary schools with 29.7 percent of their students eligible for free or reduced- 
price lunches were classified as higher poverty. Eor middle and high schools, the poverty 
thresholds were 24.3 percent and 15.9 percent, respectively. 

NETTS researchers obtained teacher rosters for the 975 schools within the districts 
selected for the NETTS district survey. To be eligible for the teacher sample, a teacher had to 
be teaching at the same school in the school year before survey administration (i.e., teachers 
new to the school were excluded). Teachers who did not teach core academic subjects also 
were omitted from the sample. Targets of four teachers from each of the schools were 
randomly selected for the teacher sample. The final teacher sample in 2007 consisted of 1,779 
teachers from 865 schools. 

Response rates were 94 percent for the district survey in 2007 and 85 percent for the 
teacher survey. Sampling weights were applied to the teacher data to obtain nationally 
representative estimates. The district and teacher surveys were initially administered during 
the 2004-05 school year to the same sample of districts and schools that provide the basis of 
comparison between 2007 and 2005 for the teacher survey data. In fall 2005, 6,017 teachers 



Districts were sampled from among the 12,483 districts that received federal Enhancing Education Through 
Technology (EETT) funds in 2003 and from an additional 2,239 districts that had not received EETT funds. 
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were surveyed with a response rate of 82 percent; the larger sample was designed to provide 
robust, school-level estimates of technology use. For additional information on the 
comparisons between 2007 and 2005 teacher survey responses, see Teachers’ Use of Student 
Data Systems to Improve Instruction: 2005 to 2007 (U.S. Department of Education 2008b). 



59 




Appendix B 



Scenario Exhibits, Items, Rubrics, Item Score Distribution and Mean Scores 
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Scenario A 



Now I’m going to describe a hypothetical situation and a computer-based student data system to you and 
I’d like to see what kind of information you’d like to get from the system. 

Suppose it’s January 2007 and you’re one of the fourth-grade teachers in a school that was surprised by 
fourth graders’ relatively low performance on the state reading test last year (spring 2006). Your principal 
has encouraged you to use student data to gain insights into how you can get higher Grade 4 
achievement this year. 

The data system available to you [show Figure A screen mockup] contains data on both current (2006- 
07) fourth graders and last year’s (2005-06) fourth graders (show Student Groups table, pointing to 
student groups as you name them) as well as other student groups. For each student, the data system 
has scores on last spring’s state reading test and scores on a district test given in the fall (point to 
relevant cells in Student Variables table). It also has other information about students that can be used to 
create subgroups within a grade if you want to see how different subgroups compare. For example, 
ethnicity, gender, and whether the student is eligible for free or reduced price lunch (FRPL). 
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Figure A: 
Student Groups 



2005-06 Third graders 


2005-06 Fourth graders 


2005-06 Fifth graders 


2006-07 Third graders 


2006-07 Fourth graders 


2006-07 Fifth graders 



Student Variables 



Ethnicity 


2005-06 Grade 3 state spring reading achievement score 


Gender 


2005-06 Grade 4 state spring reading achievement score 


Free or reduced-price lunch applicant 


2005-06 Grade 5 state spring reading achievement score 


Year entered district 


2006-07 Grade 3 district fall reading test 


Grade 3 teacher 


2006-07 Grade 4 district fall reading test 


Grade 4 teacher 


2006-07 Grade 5 district fall reading test 



Assessment System Output 



STUDENT BACKGROUND AND ASSESSMENT INFORMATION a Home 



Student Information 



Students 



FILTER BV VALUE: 

Gender ^Add filter) 

FILTER Bi VALUE: 

FRLP (Add filter) 

FILTER BY VALUE: 

(Add filter) 

FILTER BY VALUE: 



^Add filter^ 



Choose Student Group to Summarize: 



2005- 2006 Grade 32 Grade 4 _ Grade 52 

2006- 2007 Grade 3_ Grade 4^ Grade 5 _ 



Student Variables 
6 Entries Gender Ethnicity 


FRLP 


Year 

Entered 


Grade 3 
Teacher 


Grade 4 
Teacher 


200S-06 Grade 3 
language arts 
grade semester 1 


200S-06 Grade 3 
language arts 
grade semester 2 


2005-06 
Grade 3 
spring read 
achieve 


2006-07 
Grade 4 
district fall 
reading 


Jimmy 

Sampson 


M 


White 


Yes 


2004 


Simpson 


Kennison 




462 


463 




436 




430 


Lisa 

Patrick 


F 


White 


No 


2003 


Thompson 


Kennison 




481 


507 




448 




441 


Michael 

Scott 


M 


African 

Am 


No 


2003 


Thompson 


Ruiz 




472 


452 




430 




438 


Sally 

Rosen 


F 


White 


Yes 


2002 


Louise 


Hon 




430 


507 




436 




481 


Sofia Fong 


F 


Asian/Pac 

Island 


No 


2003 


Simpson 


Kennison 




448 


467 




472 




442 


Tina Smith 


F 


African 

Am 


Yes 


2004 


Thompson 


Kennison 




462 


317 




441 




436 


Tommy 

Kim 


M 


Asian/Pac 

Island 


Yes 


2002 


Louise 


Kennison 




438 


463 




481 




334 


fotals 


6F/5M 




6 FRLP 






Average 


451 


429 




448 




438 
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A.1 . So now in January 2007, what specific data would you want to get from this system to help you 
decide how to improve your fourth graders’ performance? {Follow-up probes.) What data would 
you want to see first? Tell me what group of students and what test score or student 
characteristic you would like to see for those students. 

Rubric: 

1 : Teacher picks a logical group (either 2005-06 fourth graders or 2006-07 fourth graders) AND 
selects a logical measure for that group (either Spring 2006 Grade 4 state test scores or Fall 
2006 Grade 5 district fall test scores for 2005-06 fourth graders OR either Spring 2006 Grade 3 
state scores or Fall 2006 Grade 4 district scores for 2006-07 fourth graders. 

0.5: Teacher identifies group and scorer can infer measure based on teacher response.; OR 
Teacher picks a logical group (either 2005-06 fourth graders or 2006-07 fourth graders); OR 
Teacher selects a logical measure for that group (either Spring 2006 Grade 4 state test scores or 
Fall 2006 Grade 5 district fall test scores for 2005-06 fourth graders OR either Spring 2006 
Grade 3 state scores or Fall 2006 Grade 4 district scores for 2006-07 fourth graders. 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 


0 


0.5 


1 


85 


26 (.31) 


23 (.27) 


36 (.42) 


.56 



A.2. Would you like to make any other queries of the data system? Are there other kinds of data you 
would like to have to help improve this year’s fourth graders achievement that you don’t see 
represented in this system? 

No Rubric: Coded only, not scored 
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Scenario B 



This (show Figure B) is the kind of data table that some student data systems produce. I’m going to ask 
you to find some information from the table and then I’d like for you to tell me on a scale of 1 to 10, with 
10 being extremely difficult, how hard or easy it was to find the information in the table. Ready? 



Figure B 

2005-06 Score Levels — English Language Arts (ELA) 



Language Arts 
Elementary 






Number of 


Percent 
of Tested 


Mean 




Number Students at 
Each Proficiency Level 




Grade Gender 


Ethnicity 


T ested 


Students 


Score 


Below 

Basic 


Basic 


Proficient 


Advanced 




African American 


18 


26% 


439 


5 


7 


5 


1 




Aslan/Pac Islander 


1 


1% 


610 


0 


0 


0 


1 


Female 


Latino 


17 


24% 


428 


5 


6 


5 


1 




White 


34 


49% 


449 


4 


13 


11 


6 




Total Female 


70 


100% 


444 


14 


26 


21 


9 


3 


African American 


18 


23% 


436 


6 


6 


5 


1 




Aslan/Pac Islander 


2 


3% 


452 


0 


1 


0 


1 


Male 


Latino 


31 


40% 


430 


8 


7 


14 


2 




White 


27 


35% 


448 


6 


11 


7 


3 




Total Male 


78 


100% 


438 


20 


25 


26 


7 




African American 


18 


24% 


441 


3 


8 


5 


2 




Aslan/Pac Islander 


2 


3% 


462 


1 


0 


1 


0 


Female 


Latino 


36 


47% 


436 


8 


12 


12 


4 




White 


20 


26% 


472 


2 


7 


8 


3 




Total Female 


76 


100% 


447 


14 


27 


26 


9 


4 


African American 


16 


23% 


442 


2 


8 


5 


1 




Aslan/Pac Islander 


0 


0% 


NA 


0 


0 


0 


0 


Male 


Latino 


29 


42% 


438 


5 


12 


10 


2 




White 


24 


35% 


456 


3 


13 


5 


3 




Total Male 


69 


100% 


445 


10 


33 


20 


6 




African American 


19 


26% 


463 


2 


6 


8 


3 




Aslan/Pac Islander 


1 


1% 


317 


1 


0 


0 


0 


Female 


Latino 


34 


47% 


452 


4 


14 


10 


6 




White 


19 


26% 


507 


1 


6 


7 


5 




Total Female 


73 


100% 


467 


8 


26 


25 


14 


5 


African American 


17 


23% 


449 


2 


6 


6 


3 




Aslan/Pac Islander 


3 


4% 


560 


0 


0 


1 


2 


Male 


Latino 


34 


46% 


448 


7 


13 


11 


3 




White 


20 


27% 


468 


3 


6 


8 


3 




Total Male 


74 


100% 


467 


12 


25 


26 


11 
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B.1 . What was the mean (or average) scale score for the Latino fifth-grade girls who took the test? 
How easy was it to find this information in the table? 

Rubric: 

1:452 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




49 


5 (.10) 


N/A 


44 (.90) 


.90 



B.2. Which student group had the highest average or mean ELA scale score in Grade 4? How easy 
was it to find this information in the table? 

Rubric: 

1 : “White females” or “white girls” 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




49 


6 (.12) 


N/A 


43 (.88) 


.88 



B.3. How many African American third-grade boys took the test? How easy was it to find this 
information in the table? 

Rubric: 

1: 18 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




48 


0 


N/A 


48 (1.00) 


1.00 



B.4. Overall, based on the Grade 3 data in this table, would you say that there was a difference 
between boys and girls in ELA test performance? 

No Rubric: Coded oniy, not scored 



67 







Scenario C 



This {show Figure C) is the kind of data chart that some student data systems produce. I’m going to ask you to find some 
information from the graph and then I’d like for you to tell me on a scale of 1 to 10, with 10 being extremely difficult, how 
hard or easy it was to find the information in the chart. Ready? 



Figure C 
ELA Bar Graph 



English Language Arts Achievement 




GRADE 3 



GRADE 4 



GRADE 5 

n = Number of test takers 





C.1 (II). Looking at this graph, can you find the average (or mean) English Language Arts scale score for 
the Latino fifth-grade girls who took the test? How easy was it to get this information from the 
chart? 

Rubric: 

1 : Any answer from 449 to 455 

0: Incorrect; Teacher does not know; Scorer cannot conclude whether answer is correct or 
incorrect 



^ Teachers 


Score Frequency (Proportion) 




0 


0.5 


1 


97 


25 (.26) 


N/A 


72 (.74) 



C.2. Can you find the mean score for Asian/Pacific Islander fourth-grade boys who took the test? 
How easy was it to get this information from the chart? 

Rubric: 

1 : “No” or “none took the test”; “There’s no data.” 

0: Incorrect or “There’s no information.”; Teacher does not know.; Scorer cannot conclude 
whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




97 


30 (.31) 


N/A 


67 (.69) 


.69 



C.3. Which student group had the highest average ELA score in Grade 4? Question: How easy was 
it to get this information from the chart? 

Rubric: 

1 : “White females” or “white girls” 

0.5: “White”; “orange bar for white” 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




95 


4 (.04) 


10 (.11) 


81 (.85) 


.91 



C.4. How many African American third-grade boys took the test? How easy was it to find this 
information in the table? 

Rubric: 

1:18 

0: Incorrect; Teacher does not know; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




95 


10 (.11) 


0 


85 (.89) 


.90 



69 








Scenario D 



This (show Figure D) is the kind of data table that some student data systems produce. I’m going to ask 
some questions about how you would interpret the data in this table. 



Figure D 

2005-06 Score Levels — English Language Arts (ELA) 



English Language Arts 
Hamilton Elementary 








Number 

of 

Students 


Percent 

of 


Mean 




Number Students at 
Each Proficiency Levei 


Grade Gender 


Ethnicity 


Tested 


Tested 

Students 


Score 


Below 

Basic 


Basic 


Proficient 


Advanced 




African American 


18 


26% 


439 


5 


7 


5 


1 




Aslan/Pac Islander 


1 


1% 


610 


0 


0 


0 


1 


Female 


Latino 


17 


24% 


428 


5 


6 


5 


1 




White 


34 


49% 


449 


4 


13 


11 


6 




Total Female 


70 


100% 


444 


14 


26 


21 


9 


3 


African American 


18 


23% 


436 


6 


6 


5 


1 




Aslan/Pac Islander 


2 


3% 


452 


0 


1 


0 


1 


Male 


Latino 


31 


40% 


430 


8 


7 


14 


2 




White 


27 


35% 


448 


6 


11 


7 


3 




Total Male 


78 


100% 


438 


20 


25 


26 


7 




African American 


18 


24% 


441 


3 


8 


5 


2 




Aslan/Pac Islander 


2 


3% 


462 


1 


0 


1 


0 


Female 


Latino 


36 


47% 


436 


8 


12 


12 


4 




White 


20 


26% 


472 


2 


7 


8 


3 




Total Female 


76 


100% 


447 


14 


27 


26 


9 


4 


African American 


16 


23% 


442 


2 


8 


5 


1 




Aslan/Pac Islander 


0 


0% 


NA 


0 


0 


0 


0 


Male 


Latino 


29 


42% 


438 


5 


12 


10 


2 




White 


24 


35% 


456 


3 


13 


5 


3 




Total Male 


69 


100% 


445 


10 


33 


20 


6 



70 



D.1 . Suppose you’re working with third-grade teachers at this school and they’re interested in 

examining how their students performed in terms of the language skills measured on this test. 
What does the data in this table indicate about whether boys and girls performed differently in the 
third grade? [Get open response] 

No Rubric: Coded only, not scored 



D.2. OK. Now I’m going to read a series of statements that people might make about different aspects 
of the Grade 3 data in this table. I’d like you to tell me for each statement whether you agree or 
disagree and the reasons why. Remember to think aloud as you’re deciding on your answer. 

a. A majority of third graders at this school have not achieved proficiency in ELA as measured by this test. 

Rubric: 

1: Agree [57% are less than proficient; 85/148 = (14+26+20+25)/(70+78)] 

0.5: Agree (no explanation) 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




97 


16 (.16) 


2 (.02) 


79 (.81) 


.82 



b. In Grade 3, boys were more likely than girls to score Below Basic on this assessment. 

Rubric: 

1 : Agree 25.6% boys > 20% girls (20/78 > 14/70) 

0.5: Agree (20 boys > 14 girls); Agree (no explanation) 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




96 


18 (.19) 


47 (.49) 


31 (.32) 


.58 



c. Of those students who scored Below Basic in Grade 3, most were Latino. 

Rubric: 

1 : Disagree {Less than halfot all of the students who scored Below Basic in Grade 3 are Latino.) 
0.5: Disagree with no explanation 

0: Agree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




96 


71 (.74) 


1 (.01) 


24 (.25) 


.26 



71 







D.3. Now let’s assume that you’re a fourth-grade teacher and these Grade 4 data are for mid-year performance on 
a benchmark test. If there is a particular group of fourth graders you think will be most likely to have trouble 
scoring basic or above on the state test at the end of the year, could you point out their data in this table? [if 
appropriate probe with one of the following] Which group would you be concerned about and what data 
trigger that concern? OR Why don’t you think the subgroup data in the table point to a priority need for any 
particular subgroup? 

Rubric: 

1 : Those scoring Below Basic 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 

incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




96 


86 (.90) 


5 (.05) 


5 (.05) 


.08 



Note: 

Item responses did not correlate with those to other Data Interpretation items. Scores not included in 
analyses. 

D.4. Now let’s go back to the Grade 3 data. Remember that these are for last year’s third graders. If there have 
been no major changes in the school’s student body, teachers, or curriculum would you expect that: 

a. On the basis of last year’s test scores, girls can be expected to score higher than boys when this test is 
given to this year’s third graders. 

Rubric: 

1 : Disagree [Grade 3 girls and boys scored about the same, e.g., (21+9)/70 = (26+7)/78j 
0.5: Disagree with no explanation 

0: Agree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




93 


44 (.47) 


0 


49 (.53) 


.53 



b. These data suggest that next year’s third-grade Asian/PI girls will score better than other third graders on 
this test. 

Rubric: 

1 : Disagree (Cannot predict next year’s scores based on the one Grade 3 Asian/PI student 
represented in the table) 

0.5: Disagree with no explanation 

0: Agree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




93 


39 (.42) 


2 (.02) 


52 (.56) 


.57 
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Scenario E 



This (show Figure £) is a different kind of data display — a bar graph of Grade 4 mathematics achievement separated into two components 
(computation and problem solving) as well as their total, for a school and its district for each of three years. Again, I’m going to ask you to find 
some information on the display and then tell me how easy or hard that was to do on a scale from 1 to 10 with 10 being “extremely difficult.” 



380 T 



OJ 



300 



Figure E. Trend Data Bar Graph 

Grade 4 Mathematics 

□ Oak School ■District 



2005-2006 
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E.1. What was Oak School’s average Total Math Score in 2003-04? 

Rubric: 

1 : Any specific answer from 333 to 338 

0.5: Vague, but correct answer range provided (e.g., “between 330 and 340”) 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




91 


4 (.04) 


3 (.03) 


84 (.92) 


.94 



E.2. What was the difference in the district’s total math score in 2005-06 compared with 2003-04? 



Rubric: 

1 : Any answer from 1 to 3 points; “a few points” or “up a little bit” or “about the same” or “similar” 
0.5: “exactly the same”; “the same” 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




91 


4 (.04) 


10 (.11) 


77 (.85) 


.90 



Now I’m going to ask you some questions about how you would interpret the data in this chart. 

E.3. Looking at the chart as a whole, what would this data tell you about fourth graders’ mathematics 
achievement at this school? [Get open response — No Rubric: Coded oniy, not scored] OK. Now 
I’m going to read a series of statements that people might make about the data in this graph and 
I’d like you to tell me for each one whether you agree or disagree and the reasons why. Remember 
to think aloud as you’re deciding on your answer. 

a. Oak School does better than the district as a whole in Grade 4 mathematics. 

Rubric: 

1 : Disagree (Oak School total math scores are lower than the district’s math scores.) 

0.5: Disagree with no explanation 

0: Agree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 

incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




91 


2 (.02) 


8 (.09) 


81 (.89) 


.93 
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E.3 (continued) 



b. Oak School has made some improvement in Grade 4 mathematics over this time period. 

Rubric: 

1 : Agree (Oak School total math scores are slightly higher in 2005-06 than in 2003-04.) 
0.5: Agree with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




91 


31 (.34) 


3 (.03) 


57 (.63) 


.64 



c. Relative to the district as a whole, Oak School fourth graders have been getting better in their 
problem solving skills. 

Rubric: 

1 : Agree (based on clearly Identifiable comparison of problem-solving scores for Oak School and 
the district) 

0.5: Agree (Based only on Oak School data; no clear, identifiable comparison with district); Agree 
with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




92 


38 (.41) 


10 (.11) 


44 (.48) 


.53 



d. Relative to the district as a whole. Oak School fourth graders have been getting better in their 
computation skills. 

Rubric: 

1 : Disagree (Based on clearly Identifiable comparison of computation scores for Oak School and 
the district) 

0.5: Disagree (Based only on Oak School data; no clear, identifiable comparison with district); 
Disagree with no explanation 

0: Agree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




88 


32 (.36) 


8 (.09) 


48 (.55) 


.59 
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E.3 (continued) 



e. Oak School’s progress in narrowing the Grade 4 math achievement gap with the rest of the district 
has been in problem solving rather than computation. 

Rubric: 

1 : Agree (Gap between Oak School and district scores decreases more for problem-solving than 
for computation.) 

0.5: Agree with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




89 


26 (.29) 


6 (.07) 


57 (.64) 


.67 



E.4. Suppose Oak School had started using a new mathematics program at the beginning of the 2004-05 
school year while the rest of the district continued with the old program. Looking at these data, what 
are your thoughts about the new curriculum? [Get open response — No Rubric: Coded oniy, not 
scored] Which of these statements would you agree with and why? 

a. The math program appeared to improve achievement the first year it was used but the benefit 
disappeared the next year. 

Rubric: 

1 : Disagree (Oak School is still performing better in 2005-06 than it was in 2003-04.); Agree with 
reservations (Teacher takes issue with word “disappear”. For example, “I agree, it did improve 
achievement the first year and scores are lower in the second year, but the benefit did not totally 
disappear.”) 

0.5: Disagree with no explanation 

0: Agree; Teacher does not know. Scorer cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




88 


53 (.60) 


1 (.01) 


34 (.39) 


.39 
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E.4 (continued) 

b. The new math program appeared to help students with their computation skills. 

Rubric: 

1 : Disagree (Based on comparison of computation scores for Oak School and the district; Gap 
stays relatively constant each year.) 

0.5: Disagree (look at Oak School only); Disagree with no explanation; Agree with reservations 
(i.e., “gap is pretty consistent, but the program helped a little bit”) 

0: Agree and focus only on 2 years of data (03-04 and 04-05); Agree and say that “05-06 is 
better than 03-04” but do not say “slightly better”; Other agree; Teacher does not know.; Scorer 
cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




90 


69 (.77) 


15 (.17) 


6 (.06) 


.15 



c. You can’t be sure whether the program is having an effect because there may be differences 
between the different classes of fourth graders. 

Rubric: 

1 : Agree (e.g., acknowledges that there are differences in the populations of students each year) 
0.5: Agree with no explanation 

0: Disagree (e.g., Program will have an effect regardless of the student population involved.) ; 
Teacher does not know.; Scorer cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




90 


22 (.24) 


20 (.22) 


48 (.53) 


.64 



Note: 

Item responses did not correlate with those to other Data Interpretation items. Scores not included in 
analyses. 
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E.4 (continued) 

d. Scores move around from year to year, but the new math program appears promising and should 
be monitored for more years. 

Rubric: 

1 : Agree (e.g., some indication that program benefits students; need more than 2 years of data to 
determine if a program is working) 

0.5: Agree with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




91 


10 (.11) 


15 (.16) 


66 (.73) 


.80 



E.5. Are there other issues or possible explanations that should be taken into account as well? 

No Rubric: Coded oniy, not scored 
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SCENARIO F 



Suppose you’re in a meeting to discuss 2005-06 reading data from the state assessment for your 
school’s third grade and they hand out this data display. (Show Figure F.) 



Figure F 
Histogram 



Grade 3 



45% 

40% 

35% 

30% 

25% 

20 % 

15% 

10 % 

5% 

0% 

Below Basic Basic Proficient Advanced 




Level 
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F.1v1. Based on this chart, what percentage of the school’s third graders have achieved proficiency? 



Rubric: 

1 : Any answer from 44 to 46 

0.5: Evidence that correct math is being used, but no final answer (e.g., “I need to add 38% and 
6 %.”) 



0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




47 


28 (.60) 


1 (.02) 


18 (.38) 


.39 



F.1v2. Based on this chart, what percentage of the school’s third graders were less than Proficient in 
reading? 

Rubric: 

1 : Any answer from 64 to 66 

0.5: Evidence that correct math is being used, but no final answer (e.g., “I need to add 23% and 
42%.”) 

0: Incorrect; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




49 


12 (.24) 


6 (.12) 


31 (.63) 


.69 



F.2. One of your colleagues, after looking at these data says, “There’s something wrong with this chart.” 
Would you agree? Why or why not? 

Rubric: 

1 : If agrees AND says that “numbers don’t add to 100” or “numbers add to more than 100” 

0: Incorrect (Can’t find anything wrong; Identifies something else as “wrong”); Teacher does not 
know.; Scorer cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




96 


42 (.44) 


N/A 


54 (.56) 


.56 
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Scenario G 



Suppose you’re teaching in a district that requires students to attain eighth-grade proficiency in 
mathematics in order to enroii in Aigebra i in high schooi. Looking at students’ performance the 
preceding year, you found the resuits in this tabie {show Figure G) for the Latino and African 
American students who make up your schooi’s entire student body. 

• A score of 65% on the district math test is considered proficient, and 

• A schooi is considered “iow performing” if iess than 50% of students in any student 
subgroup reach proficiency. 



Figure G 

2005-06 Achievement in Grade 8 Mathematics 



Group 


Number of Students 


Mean Math Score 


Percentage 

Proficient 


Number Proficient 


Latino 


228 


67.5 


61 


139 


African American 


31 


66 


48 


15 



81 



G.1o. What do these data tell you about how well students are doing at your school? 



Rubric: 

1 : Any correct observation or inference from the table, including “About 60% of Latino students 
are proficient.” “A smaller proportion of African American than of Latino students are proficient.” 
“The average score in the school is just a little above the proficiency criterion.” 

0: 1-2 correct general statements, but clearly is not reading the table correctly (e.g., “Latinos are 
doing better than African Americans, but both groups are not proficient.”); Other Incorrect (Do not 
reference anything in the table, very vague); Teacher does not know.; Scorer cannot conclude 
whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




82 


24 (.29) 


N/A 


58 (.71) 


.71 



G.1. Now I’m going to read you some statements again and I’d like you to tell me whether you agree or 
disagree with each and why. 

a. The majority of our eighth graders are proficient in eighth-grade math. 

Rubric: 

1 : Agree [59% = 154/259 = (139+15)/(228+31)] 

0.5: Agree with no explanation; Agree with incorrect reasoning; Disagree with different definition 
of majority (e.g., says 59% is not a majority; may state correct numbers involved in calculation, 
but still disagree) 

0: Other Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




85 


16 (.19) 


41 (.48) 


28 (.33) 


.57 



b. Our school is not getting enough African American students to proficiency, but Latino students are 
meeting the required performance standard. 

Rubric: 

1 : Agree (61% for Latinos is greater than the 50% requirement; 48% for African Americans is less 
than the 50% requirement) 

0.5: Agree with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




86 


30 (.35) 


8 (.09) 


48 (.56) 


.60 



82 







G.1 (continued) 



c. Our school is classified as low performing based on these mathematics scores. 

Rubric: 

1 : Agree (African American subgroup is less than 50% proficient.) 

0.5: Agree with no explanation 

0: Disagree (e.g., “59% of all students are proficient.”); Agree based on 65% criteria (e.g., “Both 
groups are below 65% proficient); Other agree with incorrect reasoning; Teacher does not know.; 
Scorer cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




86 


25 (.29) 


7 (.08) 


54 (.63) 


.67 



G.2. What actions should your school consider to avoid being labeled “low performing” in the coming 
year? [after open response] Which of these statements do you agree with based on these data? 
(Explain your answer for each.) [For each statement, give teacher an index card with the 
statement on it to keep handy as he/she iooks at the data.] 

a. This year all eighth graders should get more intensive instruction in mathematics. 

b. This year all African American students should get more intensive instruction in mathematics. 

Rubric 

1 : Gives same answer for G.2a and G.2b with appropriate explanation 

0.5: Same answer for G.2a and G.2b, but no explanation for one of the items; Gives the same 
answer for G.2a AND G.2b with no explanation for either item 

0: Gives the same answer for a and b with an incorrect rationale (“I disagree because the table 
doesn’t show all students in the school.”; “I agree because all of the students are not up to the 
proficient level.”); Gives different answers for G.2a and G.2b; Teacher does not know.; Scorer 
cannot conclude whether answer is correct or incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




79 


25 (.32) 


20 (.25) 


34 (.43) 


.56 
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G.2 (continued) 



c. Teachers should obtain a detailed breakdown of last year’s test results by Item or content 
standard. 

Rubric 

1 : Agree (with reasonable explanation about why having detail is important) 

0.5: Agree with no explanation 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




77 


0 


20 (.26) 


57 (.74) 


.87 



d. Our eighth-grade mathematics program isn’t necessarily “broken,” there’s a good chance that 50% 
or more of African American students will meet the proficiency requirement this year. 

Rubric: 

1 : Agree (e.g., acknowledges that sample size is small and would be possible for 1-2 students to 
do better thus bumping the percent proficient over 50%; states that 48% is close to 50% and 
would be possible to get to 50% this year) 

0.5: Agree with no explanation (no detail, missing explanation regarding w/?y African Americans 
could meet 50% level) 

0: Disagree; Teacher does not know.; Scorer cannot conclude whether answer is correct or 
incorrect. 



^ Teachers 


Score Frequency (Proportion) 


M 




0 


0.5 


1 




79 


27 (.34) 


10 (.13) 


42 (.53) 


.59 
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Scenario H 



Now I’m going to ask you to think about some data from a hypothetical classroom. 

Suppose you’re teaching mathematics and at the end of a unit on measurement, you gave a 100-point 
test on measurement concepts and skills and your students obtained the scores shown in this class list. 
[Show Figure H.] You know that students from this school have had trouble with measurement items on 
the state test in previous years, and you’re wondering whether you need to do more teaching in this area 
or can move on to the next topic. You takes these scores into the teachers’ lounge and ask colleagues to 
take a look. When they ask about the test you explain that you designed it so that if a student gets a 70% 
or better on it, you are really quite confident that he or she understands the concepts. When a student’s 
score is lower than that, you feel there is something they still don’t understand. 

One of your colleagues pulls out his calculator and shows that the mean for these scores is 70.5. “The 
mean score is greater than 70. You’ve done your job. Move on! There’s lots more math to cover.” 



Figure H 

Classroom Data Set 



Student 

Number 


Score 


1 


95 


2 


65 


3 


85 


4 


55 


5 


90 


6 


70 


7 


100 


8 


80 


9 


65 


10 


80 


11 


65 


12 


90 


13 


65 


14 


55 


15 


45 


16 


15 


17 


70 


18 


65 


19 


75 


20 


85 


21 


65 



85 



H.1 . Do you agree with this colleague? Why or why not? 

No Rubric: Coded only, not scored 

H.2. What do you think would happen if you gave the same class of students another test on 
measurement the next day? 

No Rubric: Coded only, not scored 

H.3. If you were the teacher, what would you do? 

No Rubric: Coded only, not scored 
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Appendix C 

Scenario Deveiopment and Reiiabiiity Information 
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Data Scenario Development 

Because of the project’s interest in getting at multiple, distinct facets of teachers’ 
use of data (question posing, data location, data comprehension, data interpretation and 
data use), an essentially criterion-referenced approach was taken to the development of 
the data scenario assessment items. The data scenarios were developed for this study with 
the advice and review of experts in data-informed decision making and assessment. The 
experts judged that the items represented the intended constructs. In addition, the 
scenarios were pilot-tested to make sure that they were comprehensible and tapped the 
intended skills and concepts. The mean scores for the individual items are displayed in 
Exhibit C-1. 
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Exhibit C-1 

Frequency Distributions, Mean Scores and Reliabilities for Scored Items 



Item ID 


n 

(Teachers) 


Score Frequency 


M 

(SE) 


Alpha 


0 


0.5 


1 


Form 

1 


Form 

2 


Form 

3 


Question Posing (1 item) 


A.1 


85 


26 


23 


36 


.56 

(.046) 


.469 


N/A 


.589 


Data Location (9 items) 


F.1v1 


47 


28 


1 


18 


.39 

(.071) 


.520 


N/A 


N/A 


F.1v2 


49 


12 


6 


31 


.69 

(.062) 


N/A 


.461 


N/A 


C.2 


97 


30 


N/A 


67 


.69 

(.047) 


N/A 


.450 


.566 


C.1 


97 


25 


N/A 


72 


.74 

(.045) 


N/A 


.493 


.604 


C.3 


95 


4 


10 


81 


.91 

(.025) 


N/A 


.481 


.602 


B.2 


49 


6 


N/A 


43 


.88 

(.047) 


.478 


N/A 


.374 


B.1 


49 


5 


N/A 


44 


.89 

(.032) 


.466 


N/A 


.175 


C.4 


95 


10 


0 


85 


.90 

(.044) 


N/A 


.451 


.568 


E.1 


91 


4 


3 


84 


.94 

(.023) 


.471 


N/A 


.612 


Data Comprehension (16 items) 


E.4b 


90 


69 


15 


6 


.15 

(.031) 


.475 


N/A 


.589 


D.2b 


96 


18 


47 


31 


.57 

(.036) 


N/A 


.414 


.595 


G.la 


85 


16 


41 


28 


.57 

(.038) 


N/A 


.458 


.610 


E.4a 


88 


53 


1 


34 


.39 

(.052) 


.502 


N/A 


.564 


E.3c 


92 


38 


10 


44 


.53 

(.049) 


.378 


N/A 


.526 


E.3d 


88 


32 


8 


48 


.59 

(.050) 


.469 


N/A 


.570 


F.2 


96 


42 


N/A 


54 


.56 

(.051) 


.479 


.414 


N/A 


G.lb 


86 


30 


8 


48 


.60 

(.050) 


N/A 


.434 


.595 


E.3b 


91 


31 


3 


57 


.64 

(.050) 


.469 


N/A 


.564 
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Exhibit C-1 (continued) 

Frequency Distributions, Mean Scores, and Reliabilities for Scored Items 



Item ID 


n 

(Teachers) 


Score Frequency 


M 

(SE) 


Alpha 


0 


0.5 


1 


Form 

1 


Form 

2 


Form 

3 


Data Comprehension (continued) 


G.1c 


86 


25 


7 


54 


.67 

(.049) 


N/A 


.455 


.562 


E.3e 


89 


26 


6 


57 


.67 

(.048) 


.496 


N/A 


.596 


G.1o 


82 


24 


58 


82 


.71 

(.051) 


N/A 


.455 


.605 


E.4d 


91 


10 


15 


66 


.81 

(.036) 


.494 


N/A 


.586 


D.2a 


97 


16 


2 


79 


.82 

(.038) 


N/A 


.456 


.583 


E.2 


91 


4 


10 


77 


.90 

(.026) 


.496 


N/A 


.599 


E.3a 


91 


2 


8 


81 


.93 

(.021) 


.474 


N/A 


.591 


Data Interpretation (5 items) 


D.2c 


96 


71 


1 


24 


.26 

(.044) 


N/A 


.434 


.567 


G.2ab 


79 


25 


20 


34 


.56 

(.048) 


N/A 


.471 


.587 


D.4a 


93 


44 


0 


49 


.53 

(.052) 


N/A 


.444 


.603 


G.2d 


79 


27 


10 


42 


.59 

(.052) 


N/A 


.480 


.593 


D.4b 


93 


39 


2 


52 


.57 

(.051) 


N/A 


.480 


.558 


Data Use (1 item) 


G.2c 


77 


0 


20 


57 


.87 

(.025) 


N/A 


.471 


.573 



Note: There were three forms of the data scenario interview and each item appeared on two of the 
three forms. 
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Reliability and Scores for Data Scenario Questions 

For purposes of analysis, those questions on the data scenarios with right and wrong 
answers were considered to be assessment items. In a case such as the present one, in 
which the intent is to measure performance relative to a standard rather than performance 
relative to peers, professional test publishers would estimate reliability through 
techniques associated with item response theory. These procedures require several 
hundred examinees for each item, however, and thus were out of reach for a research 
project of the present scope. 

Classical test theory can be applied with fewer respondents and examines the 
relationship of each item to performance on the rest of the items. The complication in the 
current instance is the matrix sampling of items done to manage the burden imposed on 
individual teachers. There were three forms of the data scenario interview, and each data 
scenario appeared on two of the three forms. To provide some indication of reliability, 
analysts computed alpha coefficients, examining the relationship between an individual 
item and the rest of the items in the data scenario interview, for each item and each of the 
two forms of the interview that included it. These coefficients and shown in Exhibit C- 1 . 

However, the purpose of the data scenario interviews was not to produce individual 
teacher-level scores. The goal was to provide an estimation of data literacy at the school 
level. Different teachers took different assessment forms containing different subsets of 
the items, and different teachers’ total scores are not directly comparable. 

Scales were analyzed at the school rather than the individual teacher level. That is, 
the scores for all the teachers at a school who took item 1 were averaged and that mean 
score was assigned to the school; likewise for item 2, etc. Through this process, the data 
set was structured as 27 school-level records with a score for every item. 

Reliability was estimated by computing an alpha coefficient for the total score. 
Despite the small school sample and the intentional inclusion of distinct abilities in the 
assessment, the reliability for the total score was fairly high, alpha = .74. 

For all statistical comparisons reported in the body of the report, the school-level 
total score was the variable used in the analysis. 
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